TLDR: MT-DAO is a new family of optimizers designed for distributed training of large AI models. It solves the problem of slow communication by using multiple ‘momenta’ (like memory for gradients) that operate at different speeds. This allows it to maintain stable training over long communication intervals, matching the performance of fully synchronous methods while significantly reducing training time and communication overhead, especially for large language models.
Training today’s massive artificial intelligence models, especially large language models, often requires distributing the workload across many computers. This approach, known as Distributed Data Parallelism (DDP), is powerful but comes with a significant challenge: the constant need for these computers to communicate and share information, which can saturate network bandwidth and slow everything down.
A common strategy to reduce this communication bottleneck is to have each computer perform many local updates before synchronizing with others. While this “infrequent communication” method, like Local SGD, sounds promising, it often leads to a drop in performance compared to fully synchronized training, especially when using advanced optimization techniques called adaptive optimizers.
Researchers have identified the core reason for this performance gap: a “time-scale mismatch.” Standard optimizers use a kind of “memory” called momentum that moves quickly, designed for frequent updates. When updates are infrequent, this momentum decays too rapidly, failing to smooth out the training process over longer intervals. This results in noisy and less effective optimization.
To tackle this, a new family of optimizers called MT-DAO, or Multi-timescale Distributed Adaptive Optimizers, has been proposed. MT-DAO introduces a clever solution by employing multiple types of momentum: some that move slowly and others that move quickly. The slow-moving momentum acts like a long-term memory, preserving crucial training trajectory information across those long intervals between synchronizations. Meanwhile, the fast-moving momentum ensures the optimizer remains responsive to immediate changes in the training landscape.
The beauty of MT-DAO is that it provides the first theoretical guarantees for convergence in this multi-momentum, distributed setting. In its simplest form, it doesn’t even require extra memory or communication overhead, needing only one additional setting to tune.
Empirical results from pre-training large language models show that MT-DAO successfully eliminates the performance gap with fully synchronous DDP. It consistently outperforms other infrequent-communication methods in terms of model quality (perplexity) and significantly reduces the actual time it takes to train. For instance, on a 720 million parameter model, MT-DAO achieved a target performance level in 24% fewer training steps and 35% less wall-clock time compared to a standard DDP baseline.
This improved efficiency is attributed to several key benefits of MT-DAO. Firstly, its slow momentum effectively suppresses noise and retains vital information across communication rounds, leading to a more stable optimization path. Secondly, it makes the training process more resilient to infrequent communication by reducing how much the model parameters and optimizer states change between synchronizations. This means that when workers do synchronize, their models haven’t diverged too much, making the averaging step more effective.
Furthermore, MT-DAO ensures that the training paths of individual workers remain highly aligned with the overall global training direction. This “regularizing” effect of the slow momentum reduces worker drift and promotes a stable, shared optimization trajectory across all distributed machines.
Also Read:
- AMAQ: Smarter Data Compression for Efficient Large Language Model Training
- REG Optimizer: Enhancing Stability and Performance in Large Language Model Training
This breakthrough in distributed training offers a robust and practical way forward for scaling the training of foundation models, especially in environments with communication constraints, such as training across different data centers or wide geographic areas. For more details, you can read the full research paper: MT-DAO: Multi-timescale Distributed Adaptive Optimizers with Local Updates.
