TLDR: ByteRobust is ByteDance’s advanced GPU infrastructure management system designed to overcome the challenges of training large language models (LLMs) at massive scales. It employs an automated fault tolerance framework with real-time and stop-time checks, data-driven over-eviction for implicit failures, and swift recovery mechanisms like hot-updates, warm standbys, and zero-overhead checkpointing. Deployed on over 200,000 GPUs, ByteRobust has achieved 97% Effective Training Time Ratio (ETTR) by efficiently detecting, diagnosing, and recovering from various failures, significantly reducing downtime and improving training efficiency.
Training large language models (LLMs) has become a monumental task, involving tens of thousands of GPUs and spanning months. While this scale enables the creation of increasingly powerful AI, it also introduces a significant challenge: frequent failures. These can range from obvious hardware errors like CUDA failures to subtle issues like job hangs or unexpected performance drops. Such interruptions lead to considerable downtime, severely impacting the efficiency of the training process.
Recognizing these hurdles, ByteDance has developed ByteRobust, a sophisticated GPU infrastructure management system designed specifically to ensure stable and robust LLM training. The system prioritizes minimizing training interruptions, quickly diagnosing faults, and effectively tolerating failures to maintain highly efficient, continuous training.
A New Philosophy for Robust Training
ByteRobust operates on a unique set of principles. Instead of spending valuable time on precise fault localization, it focuses on rapid isolation. In a cluster with thousands of GPUs, quickly identifying and isolating a problematic machine, even if it means temporarily removing a few healthy ones, is often more efficient than a lengthy, exact diagnosis that leaves many GPUs idle. The system also acknowledges that human errors, such as bugs in continuously evolving user code, are an inevitable source of failures. ByteRobust integrates mechanisms like code rollbacks and ‘lazy updates’ to manage these. Furthermore, it emphasizes controlled and swift recovery, using techniques like ‘hot-updates’ for code changes, ‘warm standbys’ for machine replacements, and intelligent checkpointing to ensure stability during recovery.
How ByteRobust Tackles Failures
The system is divided into two main components: a control plane and a data plane. The control plane orchestrates the overall strategy, detecting anomalies, locating faults, and triggering recovery actions. The data plane, residing within each training unit (pod), continuously monitors, diagnoses, manages checkpoints, and captures real-time information.
ByteRobust employs an automated fault tolerance framework that starts with proactive real-time checks. These checks constantly monitor system health, including network status, GPU performance, and host events. If a clear issue is detected, like a GPU becoming unavailable, the system can immediately evict the problematic machine. For less obvious issues or user-space errors, it might trigger a code rollback or move to more in-depth ‘stop-time checks’.
The stop-time checks involve a hierarchical approach. The system first attempts to diagnose the issue using logs and specific tests (e.g., for network communication or GPU health). If tests pass, it might simply reattempt the training, assuming a transient fault. If problems persist, it can roll back recent user code changes. For elusive problems like Silent Data Corruption (SDC), ByteRobust uses a clever ‘dual-phase replay’ method, which involves replaying parts of the training on smaller groups of machines to pinpoint the faulty one without disrupting the entire cluster.
Handling the Unseen: Data-Driven Over-Eviction
Some of the most challenging failures are ‘implicit’ ones, such as job hangs where no logs are generated, or gradual performance declines (MFU declines) where all machines appear to slow down simultaneously. For these, ByteRobust uses ‘data-driven over-eviction’. When such a silent failure is detected, the system inspects the ‘stack traces’ (the sequence of function calls) of all internal training processes. By comparing these traces, it can identify outlier processes and, by extension, the machines or groups of machines that are behaving abnormally. To ensure a quick recovery, ByteRobust might ‘over-evict’ an entire parallel group of machines, even if only one or two are truly faulty, prioritizing speed over pinpoint accuracy.
Controlled and Swift Recovery
Once a fault is identified and isolated, ByteRobust focuses on getting the training back on track as quickly as possible. For code or data adjustments, it uses an ‘in-place hot-update’ mechanism, which allows modifications without requiring a full job restart or rescheduling new machines. This is significantly faster than traditional methods.
To replace evicted machines, ByteRobust maintains a pool of ‘warm standby machines’. These machines are pre-provisioned, self-checked, and kept in a low-power state, ready to be activated instantly. This eliminates the time-consuming process of scheduling and initializing new machines from scratch. The number of standby machines is dynamically adjusted based on historical failure rates.
Finally, ByteRobust implements an ‘over-eviction-aware checkpointing’ system. Instead of relying on slow remote storage, it saves checkpoints (the model’s progress) to local CPU memory and disk. It also uses a ‘cross-parallel group backup strategy’, ensuring that backups are stored on machines outside the same parallel group, making them resilient even if an entire group is over-evicted. This asynchronous checkpointing process is designed to have near-zero overhead, allowing for frequent saves without impacting training performance.
Also Read:
- ShadowServe: Boosting LLM Performance with SmartNIC-Powered KV Cache Management
- RLinf: Smarter Scheduling for Faster Reinforcement Learning
Real-World Impact and Future Outlook
ByteRobust has been deployed on ByteDance’s production GPU clusters, managing over 200,000 GPUs. It has demonstrated remarkable effectiveness, achieving an impressive 97% Effective Training Time Ratio (ETTR) for a three-month training job on 9,600 GPUs. The system has successfully identified and resolved tens of thousands of explicit and implicit failures, significantly reducing unproductive time. Its hot-update and warm standby mechanisms have accelerated recovery times by over 10 times compared to traditional methods, and its checkpointing system adds less than 0.9% overhead to training. This continuous optimization has also led to substantial improvements in Model FLOPs Utilization (MFU) over time.
While ByteRobust represents a significant leap in robust LLM training, challenges remain. The rapid evolution of GPU hardware means diagnostic tools often lag, making root cause analysis difficult. The system also experiences ‘false positives’, sometimes over-evicting healthy machines to ensure rapid isolation. Silent Data Corruption (SDC) continues to be a critical, hard-to-detect issue, requiring further research into more efficient detection and isolation techniques. Despite these challenges, ByteRobust provides a robust foundation for scaling LLM training to unprecedented levels. You can read more about this work in the research paper: Robust LLM Training Infrastructure at ByteDance.
