TLDR: EvoSLD is a new automated framework that uses evolutionary algorithms guided by Large Language Models (LLMs) to discover neural network scaling laws. It co-evolves mathematical expressions and their optimization routines, distinguishing between scaling and control variables. Tested on five real-world scenarios, EvoSLD rediscovered human-derived laws and surpassed them in accuracy in others, significantly outperforming traditional methods. The framework highlights the importance of co-evolving optimization strategies for better parameter fitting and promises to accelerate AI research by automating complex law discovery.
In the rapidly evolving field of artificial intelligence, particularly with the rise of large language models (LLMs), understanding how neural network performance changes with factors like model size, dataset size, and computational resources is crucial. These predictable relationships are known as scaling laws. Traditionally, uncovering these fundamental mathematical relationships has been a painstaking process, demanding significant human expertise and countless hours of manual experimentation.
A new automated framework, called EvoSLD, is set to change this. EvoSLD, which stands for Evolutionary Scaling Law Discovery, introduces an innovative approach that combines evolutionary algorithms with the power of Large Language Models (LLMs). This framework is designed to automatically discover these elusive scaling laws by co-evolving both the mathematical expressions that define the laws and the optimization routines needed to fit them to data.
What makes EvoSLD unique is its ability to differentiate between ‘scaling variables’ (like the number of training tokens or model parameters, which are actively changed in experiments) and ‘control variables’ (such as model architecture or optimizer, which remain constant within an experiment but vary across different settings). This distinction allows EvoSLD to search for a single, universal mathematical form that can apply across diverse experimental conditions, with only its coefficients needing adjustment for each specific setting. This mirrors how scientists typically approach discovering general principles.
The core of EvoSLD operates through an evolutionary search process. It starts with a basic set of candidate laws and their fitting methods. In each cycle, the system selects promising candidates, and an LLM then proposes intelligent modifications to their code – either to the mathematical expression or the optimization method. These modified candidates are then evaluated on a development dataset, and the best-performing ones are kept and refined in subsequent iterations. This continuous refinement allows EvoSLD to efficiently explore a vast space of potential laws.
EvoSLD’s effectiveness has been rigorously tested across five real-world scenarios drawn from recent AI research. These include studies on vocabulary size, supervised fine-tuning, domain mixtures, Mixture-of-Experts (MoE) models, and data-constrained pre-training. The results are quite impressive. In two cases (vocabulary size and supervised fine-tuning), EvoSLD successfully rediscovered the exact laws that human experts had previously derived. More remarkably, in other, more complex scenarios like domain mixture and data-constrained pre-training, EvoSLD surpassed human-designed laws, achieving significantly lower fitting errors on unseen data. This suggests that for intricate systems, an automated, LLM-guided search can uncover insights that human intuition alone might miss.
A key finding from the experiments is the critical role of ‘co-evolution’ – allowing both the symbolic expression and its optimization routine to evolve together. This significantly improves performance, especially when dealing with sparse or multi-variable data, where a fixed, general-purpose optimizer might struggle. By developing specialized optimization strategies alongside the law itself, EvoSLD achieves more accurate parameter fits, making the discovered laws more practically useful for applications like model selection and determining optimal training configurations.
The research also highlights the limitations of traditional symbolic regression methods, which often fail to find meaningful relationships or produce overly complex, uninterpretable equations when applied to scaling law discovery. EvoSLD, by contrast, leverages the advanced understanding embedded in LLMs to guide its search towards plausible and well-structured laws, avoiding the pitfalls of unguided exploration.
While concerns about data leakage (where the LLM might simply recall laws from its training data) were considered, the diversity of solutions discovered by independent EvoSLD runs for the same problem suggests genuine problem-solving rather than mere memorization. The framework currently relies on passively collected data from published papers, which can be challenging for fitting multi-variable functions. Future work aims to develop more advanced, agent-based systems that can actively propose and conduct new experiments, further accelerating the scientific discovery process.
Also Read:
- MeLA: Guiding Large Language Models to Design Better Algorithms
- Optimizing Multiple Objectives: A New Approach with Language Models and Grid Guidance
EvoSLD represents a significant leap forward in automating scientific discovery. For instance, it can rediscover complex scaling laws in minutes that previously required weeks of human effort. This dramatic speed-up positions EvoSLD as an invaluable tool for researchers and engineers, particularly in the LLM industry, enabling rapid hypothesis generation, validation, and optimization of models. You can find more details about this groundbreaking work in the full research paper available at this link.
