TLDR: This paper explores practical strategies for scaling large language model (LLM) pretraining, addressing the scarcity of public information on this topic. Researchers at MIT Lincoln Laboratory detail lessons learned from training a BERT-like model on a novel binary code dataset using a supercomputing cluster. Key recommendations include extensive dataset preprocessing, duplicating datasets locally on nodes, optimizing parallel data loading, noting that network bandwidth is less of a bottleneck for data parallelism than expected, and understanding how larger models can reduce training efficiency due to memory constraints.
Large Language Models (LLMs) have become central to many natural language processing applications, showcasing impressive performance. However, training these advanced models is an incredibly resource-intensive endeavor, with leading AI companies investing billions in supercomputing infrastructure. Despite this massive investment, detailed information regarding the scaling performance and practical training considerations for these large pipelines is rarely shared publicly. This scarcity of practical recommendations makes it challenging for researchers to optimize training performance when scaling up LLMs.
This research paper aims to shed light on the LLM pretraining pipeline, focusing on critical aspects such as distributed training, managing vast datasets across hundreds of computing nodes, and effectively scaling data parallelism to fully utilize available GPU compute capacity.
Over the past few years, the primary goal for many frontier AI research companies has been to scale up LLMs. Research consistently shows that increasing model size, given sufficient data, predictably improves performance across various tasks. Naturally, larger models and datasets demand more computational resources for training. Companies like Google, OpenAI, Anthropic, and others have poured significant resources into building supercomputing capabilities to develop their cutting-edge AI models.
Interestingly, recent model releases from these companies, such as OpenAI’s GPT-4.5, Google’s Gemini 2.5, Anthropic’s Claude 3.5 Sonnet, xAI’s Grok 3, Mistral AI’s Medium 3, and DeepSeek’s R1, offer very little transparency about their underlying architectures or training processes. Even basic details like model parameter counts are often kept private, leading to speculation. This lack of public information forces other researchers to independently tackle challenges related to scaling, distributed data access, bottleneck resolution, and best practices.
In this work, the researchers at Lincoln Laboratory, Massachusetts Institute of Technology, pretrained an LLM for a novel language application. They meticulously examined scaling performance as they increased both the dataset and model size, documenting valuable lessons learned throughout the process. All experiments were conducted on a computing cluster comprising several hundred nodes equipped with Nvidia H100 GPUs, specifically the TX-GAIN system at the Lincoln Laboratory Supercomputing Center (LLSC).
The TX-GAIN cluster consists of 316 HPE compute nodes, each featuring dual AMD EPYC 9254 24-core CPUs with 768 GB of DRAM and dual Nvidia Hopper H100-NVL GPUs, each with 94 GB of HBM GPU RAM. These H100-NVL GPUs are interconnected by an NV-Link bridge on each node. The nodes access central Lustre parallel storage arrays, possess 3.8 TB of local SSD storage, and are connected via a 25-Gigabit Converged Ethernet link to a central core switch.
Also Read:
- SpikingBrain: Brain-Inspired Models for Efficient Large Language Processing
- Unveiling Language Model Learning: Tracking Linguistic Feature Development During Pretraining
Key Insights and Recommendations for Scaling LLM Pretraining:
The researchers pretrained a Bidirectional Encoder Representation for Transformers (BERT)-like encoder model on a large, novel dataset of binary code. The model was trained using a Masked Language Modeling (MLM) task. They scaled their experiments from a small 120M parameter model on a single GPU to a larger 350M parameter model distributed across 128 nodes with 256 GPUs.
Here are their practical recommendations:
1. Preprocess and Tokenize Extensively: The initial dataset of binary code functions was nearly 2 TB. After preprocessing and tokenizing, storing only the necessary training data (tokenized inputs and attention masks), the dataset size dramatically reduced to just 25 GB. This 99% reduction is crucial for managing large datasets.
2. Duplicate Datasets Locally: For smaller, preprocessed datasets, copying the entire dataset from network storage to local SSD storage on each node before training proved highly beneficial. This initial cost was worthwhile to prevent nodes from contending for network and network-attached storage resources during training, eliminating a significant bottleneck.
3. Optimize Parallel Data Loading: Even after extensive preprocessing, loading data into GPU memory remained a bottleneck. The researchers recommend gradually increasing the number of parallel data loaders until single GPU utilization stabilizes near 100%. Adding more loaders beyond this point would simply waste resources.
4. Network Bandwidth for Data Parallelism: Surprisingly, for data-parallel multi-node training, network bandwidth was not as significant a bottleneck as one might expect. The model training performance scaled roughly linearly with the number of nodes, even up to 128 data-parallel nodes. This indicates that the workload remained GPU-bound, rather than being limited by network performance during weight updates.
5. Model Size and Training Efficiency: As model size increased (while keeping the number of nodes constant), training performance decreased. This is because larger models require more GPU memory due to their increased parameter count, which in turn reduces the possible training batch size. For instance, their 120M parameter model used a batch size of 184 samples, while the 350M parameter model only managed 20. Scaling further would likely necessitate model parallelism, requiring additional optimization.
These practical recommendations, derived from real-world experience in scaling custom LLM pretraining, offer valuable guidance for other researchers navigating the complexities of large-scale model development. You can read the full research paper here: Scaling Performance of Large Language Model Pretraining.
