TLDR: NVIDIA introduces NVFP4, a novel 4-bit floating point format and a comprehensive training methodology that enables stable and accurate pretraining of large language models (LLMs) at scale. By incorporating Random Hadamard transforms, two-dimensional quantization, stochastic rounding, and selective high-precision layers, the approach allows a 12-billion-parameter model to be trained on 10 trillion tokens with performance comparable to FP8 baselines, significantly improving computational efficiency and resource utilization.
Large Language Models (LLMs) are becoming increasingly powerful, but training these advanced models demands immense computational resources, time, and energy. As models grow in size and complexity, improving pretraining efficiency is crucial for developing the next generation of even more capable LLMs.
While 8-bit floating point (FP8) training has become a standard, the move to narrower precision, such as 4-bit floating point (FP4), promises significant improvements in computational speed and resource utilization. However, quantizing models to this level of precision presents considerable challenges, including maintaining training stability, ensuring convergence, and implementing it effectively, especially for large-scale models trained over extensive token horizons.
NVIDIA has introduced a groundbreaking approach for stable and accurate training of LLMs using their novel NVFP4 format. This method integrates several key innovations to overcome the inherent difficulties of 4-bit precision training. These include the use of Random Hadamard transforms (RHT) to manage block-level outliers, a two-dimensional quantization scheme that ensures consistent representations during both the forward and backward passes, the application of stochastic rounding for unbiased gradient estimation, and the strategic incorporation of selective high-precision layers for numerically sensitive parts of the model.
To validate this new methodology, NVIDIA successfully trained a 12-billion-parameter model on an unprecedented 10 trillion tokens. This represents the longest publicly documented training run in 4-bit precision to date. The results are highly encouraging: the model trained with the NVFP4-based pretraining technique achieved training loss and downstream task accuracies comparable to an FP8 baseline. For instance, it reached an MMLU-pro accuracy of 62.58%, nearly matching the 62.62% accuracy obtained through FP8 pretraining. These findings underscore that NVFP4, when combined with this innovative training approach, marks a significant leap forward in narrow-precision LLM training algorithms.
The NVFP4 format itself is an enhancement over existing 4-bit microscaling formats like MXFP4. NVFP4 utilizes a smaller micro-block structure (16 elements compared to MXFP4’s 32), which more effectively captures local dynamic range. It also employs an FP8 scale factor format with fractional precision for more accurate microscaling and a two-level scaling strategy that combines a fine-grained FP8 scale factor with a tensor-level FP32 scale. These design choices contribute to a more precise representation of tensor values during training.
NVIDIA Blackwell GPUs provide native hardware support for NVFP4 through their Tensor Cores, delivering a two to three-fold boost in arithmetic performance and halving memory usage compared to FP8. This hardware integration is critical for realizing the practical benefits of FP4 training.
Further comparisons with MXFP4 demonstrated NVFP4’s efficiency advantage. In side-by-side experiments, NVFP4 achieved comparable loss with fewer tokens than MXFP4, indicating significant efficiency gains without sacrificing accuracy. MXFP4 required 36% more tokens to match NVFP4’s loss, highlighting the benefits of NVIDIA’s new format.
Also Read:
- Optimizing Large Language Model Training with Fine-Grained Data Management
- CoA-LoRA: Dynamic Adaptation for Quantized LLMs on Diverse Edge Devices
This research establishes the feasibility of sustained 4-bit pretraining at a multi-trillion-token scale, paving the way for faster and more efficient training of future frontier models. The code supporting Transformer Engine for NVFP4 training is now available, enabling broader adoption of this advanced precision format. For more technical details, you can refer to the original research paper: Pretraining Large Language Models with NVFP4.
