TLDR: A research paper empirically analyzes the compute requirements for developing algorithmic innovations in large language models, cataloging 36 innovations from Llama 3 and DeepSeek-V3. It finds that the compute needed for these innovations is rapidly increasing, doubling annually for significant resources. The study suggests that compute caps alone are unlikely to dramatically slow AI algorithmic progress, as even strict limits would still permit the development of a substantial portion of innovations.
A new research paper titled “Compute Requirements for Algorithmic Innovation in Frontier AI Models” by Peter Barnett delves into the often-overlooked computational demands behind the development of new AI algorithms. While it’s widely known that algorithmic advancements have significantly reduced the total compute needed to achieve a certain level of AI capability, this paper empirically investigates the compute resources required to *create* these innovations in the first place.
The study meticulously catalogs 36 pre-training algorithmic innovations found in prominent open models like Llama 3 and DeepSeek-V3. For each innovation, the author estimates both the total computational operations (FLOP) used during its development and the hardware capacity (FLOP/s) of the systems utilized. This detailed analysis provides a clearer picture of the resources that fuel algorithmic progress.
One of the key findings is that the compute requirements for developing significant algorithmic innovations are increasing rapidly, roughly doubling each year. This trend applies to both the total operations and the hardware capacity. Interestingly, this increase is happening at a faster rate than the decrease in compute cost per dollar, suggesting that the growing demand isn’t solely due to cheaper computational resources.
The paper also explores the potential impact of “compute caps” – restrictions on either the total computational operations or the hardware capacity available to AI developers. This is a crucial area for AI governance, as limiting compute is seen as a central lever for controlling AI development. The analysis suggests that, surprisingly, compute caps alone might not dramatically slow down algorithmic progress. Even stringent caps, such as limiting total operations to the compute used to train GPT-2 or capping hardware capacity to eight H100 GPUs, could still have allowed for the development of about half of the innovations cataloged in the study.
Also Read:
- Navigating AI’s Future: Technical Pathways for Halting Dangerous Development
- DeepSeek’s Advancements in Large AI Models: Efficiency, Reasoning, and Open-Source Impact
This implies that existing hardware in various regions might already be sufficient for current algorithmic innovation, potentially limiting the effectiveness of export controls on advanced AI hardware as a standalone measure to slow down rival states’ AI progress. The paper acknowledges several limitations, including focusing only on reported successful experiments (which might underestimate actual compute spent), initial research, and excluding proprietary models. However, it highlights the importance of understanding these compute dynamics for effective AI governance, suggesting that compute caps may need to be combined with other regulatory approaches to be truly effective.
