TLDR: The Ring-linear model series, including Ring-mini-linear-2.0 (16B parameters) and Ring-flash-linear-2.0 (104B parameters), introduces an efficient hybrid architecture for Large Language Models (LLMs) that combines linear and softmax attention. This design significantly reduces computational and I/O overhead in long-context scenarios, cutting inference costs by up to 90% compared to dense models and over 50% compared to previous Ring series models. Enhanced by a self-developed FP8 operator library and systematic training-inference alignment, the models achieve stable reinforcement learning and maintain state-of-the-art performance on complex reasoning benchmarks.
Large Language Models (LLMs) are becoming increasingly powerful, but their ability to handle very long texts, known as long-context reasoning, has been a significant challenge. Traditional LLM architectures, which rely heavily on a mechanism called Softmax Attention, face a major hurdle: their computational demands grow quadratically with the length of the text. This means that as the text gets longer, the resources needed to process it skyrocket, making long-context applications like advanced AI agents or complex code generation very expensive and slow.
To address this, researchers have explored Linear Attention, a more efficient alternative where computational costs increase linearly with text length. While promising, pure Linear Attention models often don’t perform as well in real-world, large-scale scenarios, especially as models become bigger and contexts get longer. This led to the idea of hybrid architectures, which combine the best of both worlds: retaining some Softmax Attention for its expressive power while leveraging Linear Attention for efficiency.
Introducing the Ring-linear Model Series
A new technical report introduces the Ring-linear model series, featuring Ring-mini-linear-2.0 and Ring-flash-linear-2.0. These models adopt an innovative hybrid architecture that seamlessly integrates linear attention and softmax attention. The goal is to drastically cut down on the input/output (I/O) and computational overhead when dealing with long texts.
The Ring-mini-linear-2.0 is a more compact model with 16 billion parameters, while the Ring-flash-linear-2.0 is a larger model boasting 104 billion parameters. Both models demonstrate remarkable efficiency gains. For instance, they can reduce inference costs to just one-tenth compared to a 32-billion-parameter dense model. Even against the previous generation of Ring models, the cost is slashed by over 50%.
Optimized for Performance and Stability
The development of the Ring-linear series involved a systematic exploration of how to best combine different attention mechanisms, leading to an optimal model structure. Beyond architectural innovations, the team also developed a high-performance FP8 operator library called ‘linghe’. This library alone improved overall training efficiency by 50%.
A critical aspect highlighted in the report is the importance of aligning training and inference processes, especially during the reinforcement learning (RL) phase. Discrepancies between how models behave during training and how they perform during actual inference can lead to instability and limit performance. The Ring-linear models benefit from a high degree of alignment between their training and inference engine operators, allowing for stable, long-term, and highly efficient optimization during RL. This ensures the models consistently achieve state-of-the-art (SOTA) performance across various challenging complex reasoning benchmarks.
Under the Hood: Key Innovations
The Ring-linear architecture is built upon a highly sparse Mixture-of-Experts (MoE) design, maximizing the use of linear attention. Key architectural choices include Grouped RMSNorm for efficient normalization, Partial Rotary Position Embedding (RoPE) for improved training loss, and a head-wise power-law decay rate for the hidden state in Linear Attention, which significantly impacts downstream task performance.
Extensive computational optimizations were also implemented. These include GPU kernel fusion, which combines multiple operations into single, more efficient steps, reducing latency and memory consumption. Furthermore, FP8 training optimization, which uses lower precision (FP8) for calculations, was carefully integrated with quantization fusion and state-aware recomputation to boost speed without sacrificing accuracy. These optimizations led to substantial improvements in both training and inference throughput, especially for longer context lengths.
The models undergo a two-stage continued pre-training process to restore and extend their capabilities, followed by post-training involving Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). The systematic training-inference alignment, addressing subtle implementation differences in modules like KV Cache, LM Head, RMSNorm, RoPE, Attention, and MoE, was crucial for achieving stable and effective RL training.
Also Read:
- Ring-1T: Unveiling the First Trillion-Parameter Open-Source Thinking Model
- Adaptive Precision for Language Models: A New Frontier in Efficiency
Impact and Future Outlook
The Ring-linear series models demonstrate strong reasoning capabilities across mathematical reasoning, agent and coding tasks, and general reasoning benchmarks. Ring-mini-linear-2.0, despite its smaller size, performs comparably to larger counterparts, while Ring-flash-linear-2.0 delivers highly competitive performance against state-of-the-art models in its class.
This research marks a significant step towards developing more efficient and capable LLMs for long-context reasoning, paving the way for more advanced AI applications. For more details, you can read the full research paper here.
