TLDR: Ling 2.0 is a new series of reasoning-focused language models, scaling up to one trillion parameters using a high-sparsity Mixture-of-Experts (MoE) architecture. It achieves 7x active-compute efficiency through innovations in model design, reasoning-oriented pre-training with Chain-of-Thought activation, adaptive post-training using Decoupled Fine-Tuning and Evolutionary CoT, and an FP8-based infrastructure. The models demonstrate superior reasoning accuracy and efficiency, establishing a new benchmark for scalable and intelligent AI.
The field of artificial intelligence continues to advance at an incredible pace, with large language models (LLMs) becoming central to modern AI. A new technical report introduces Ling 2.0, a groundbreaking series of reasoning-oriented language models designed to scale to an astonishing one trillion parameters. Developed by the Ling Team at Inclusion AI, Ling 2.0 operates on the principle that “every activation boosts reasoning capability,” aiming to create highly efficient and accurate intelligent systems.
Ling 2.0 is not just another large language model; it’s a comprehensive foundation built with coordinated innovations across several key areas. These include a novel model architecture, advanced pre-training and post-training methodologies, and robust infrastructure. The series features three “non-thinking” (instruct) models: Ling-mini-2.0 (16 billion parameters), Ling-flash-2.0 (103 billion parameters), and the colossal Ling-1T (one trillion parameters). These models demonstrate up to 7 times greater active-compute efficiency compared to their dense counterparts, setting a new standard for reasoning accuracy and computational efficiency at the trillion-parameter scale.
The Core Architecture: High-Sparsity Mixture-of-Experts
At the heart of Ling 2.0’s efficiency is its high-sparsity Mixture-of-Experts (MoE) architecture. Unlike traditional dense models where all parameters are activated for every computation, MoE models selectively activate a small subset of “experts” for each input. Ling 2.0 uses 256 routed experts, activating 8 experts plus one shared expert per token, resulting in approximately 3.5% activation. This sparse activation is crucial for achieving significant efficiency gains without sacrificing performance.
The design is guided by “Ling Scaling Laws,” derived from thousands of experiments, which ensure stable and near-optimal training for trillion-parameter models. These laws allow for reliable predictions of large-scale performance from smaller experiments, drastically cutting validation costs. The architecture also incorporates Multi-Token Prediction (MTP) as an auxiliary training objective, consistently improving performance on code and math tasks across different model scales.
Pre-training for Enhanced Reasoning
Ling 2.0’s pre-training strategy is meticulously designed to imbue the models with strong reasoning capabilities from the outset. The pre-training corpus prioritizes specialized datasets like “Ling Math” and “Ling Code,” which are tailored for mathematical reasoning and code generation. These datasets contribute to a 5-8% average gain on reasoning benchmarks. Over the 20 trillion-token pre-training process, the proportion of reasoning data is progressively increased from 32% to 46%.
A unique aspect of Ling 2.0’s pre-training is “Reasoning Pre-Activation in Mid-Training.” During this phase, the effective context window is extended, and Chain-of-Thought (CoT) data is introduced to pre-activate reasoning abilities. This strategy not only raises the ceiling on reasoning performance but also provides a more stable foundation for subsequent fine-tuning. The training also utilizes a novel “Warmup-Stable-Merge (WSM) Scheduler,” which replaces traditional learning-rate decay with checkpoint merging, offering greater flexibility and delivering 1-2% average gains across benchmarks.
Post-training for Adaptive Intelligence
The post-training phase refines Ling 2.0 into a versatile foundation model capable of deep reasoning in complex scenarios while remaining efficient for everyday tasks. This involves a three-stage methodology:
- Decoupled Fine-Tuning (DFT): This approach uses differentiated system prompts to create two distinct response modes: “Instant Response” for quick answers and “In-Depth Reasoning” for complex problem-solving. This establishes a dedicated deep-reasoning mode, preparing the model for further enhancement.
- Evolutionary Chain-of-Thought (Evo-CoT): Building on DFT, Evo-CoT instills adaptive reasoning, allowing the model to adjust its reasoning depth based on problem complexity. It uses a composite reward function that balances accuracy, dynamic length control (encouraging brevity for easy tasks and elaborate reasoning for hard ones), and formatting. A novel “Linguistic-unit Policy Optimization (LPO)” algorithm performs sentence-level policy updates, leading to smoother reward curves and greater stability in reinforcement learning.
- Group Arena Reward (GAR): For open-ended, subjective tasks, GAR employs an intra-group comparative evaluation strategy. Multiple responses from the same policy are compared in a “tournament-style” fashion by a generative reward model, reducing variance and noise in subjective quality assessment. This is complemented by “RubriX,” a fine-grained, multi-dimensional reward guideline framework for precise preference modeling.
Also Read:
- Boosting LLM Efficiency: How Token Permutation Makes Attention Sparser
- LLMs Learn to Self-Improve Through Multi-Agent Co-Evolution
Infrastructure for Trillion-Parameter Scale
Scaling to a trillion parameters requires cutting-edge infrastructure. Ling 2.0 achieves optimal training and inference efficiency through several key innovations:
- Full-scale FP8 Training: Ling 2.0 is the largest open-source model trained entirely in FP8 precision. This fine-grained quantization strategy achieves near-lossless accuracy while improving utilization and reducing memory use by over 15%.
- Heterogeneous Fine-grained Pipeline Parallelism: To mitigate pipeline bottlenecks caused by varying computational and memory demands of different model components (like MTP layers and dense layers), the pipeline parallelism framework was modified. This resulted in a 40% relative end-to-end improvement in training throughput.
- Distributed Training Framework Optimizations: Enhancements like “Intra-Node DeepEP” for optimized expert parallelism communication, fused kernels for reduced CPU overhead, and “Fast Expert Full-Recomputation” for memory savings contribute to overall performance and stability.
- Software Engineering for Foundation LLMs: The project adopted a “4C (Correct, Consistent, Complete, and Co-Design)” principle to manage the complexity of developing trillion-scale models, ensuring robust and efficient development.
The Ling 2.0 series, including Ling-mini-2.0, Ling-flash-2.0, and Ling-1T, consistently follows the Ling Scaling Law, demonstrating superior performance across a wide range of benchmarks in mathematics, coding, reasoning, knowledge, and alignment tasks. The Ling-1T model, in particular, establishes a new Pareto frontier of reasoning accuracy versus computational efficiency, proving that sparse activation, when properly aligned with reasoning objectives, enables scalable and efficient intelligence. This work provides a coherent, open, and efficient foundation for advancing future reasoning and thinking models. For more in-depth technical details, you can read the full Ling 2.0 Technical Report.
