TLDR: Kimi K2 is a 1 trillion-parameter Mixture-of-Experts large language model with 32 billion activated parameters, developed by the Kimi Team. It introduces the MuonClip optimizer for stable training on 15.5 trillion tokens and features a multi-stage post-training process including large-scale agentic data synthesis and a joint reinforcement learning stage. Kimi K2 achieves state-of-the-art performance among open-source models in agentic capabilities, coding, mathematics, and reasoning tasks, with strengths in software engineering and tool use.
The Kimi Team has unveiled Kimi K2, a groundbreaking large language model designed to push the boundaries of agentic intelligence. This new model, a Mixture-of-Experts (MoE) architecture, boasts an impressive 1 trillion total parameters with 32 billion activated parameters, positioning it as one of the most capable open-source LLMs to date, especially in areas like software engineering and autonomous agent tasks.
Innovations in Training Stability and Efficiency
A key innovation in Kimi K2 is the introduction of the MuonClip optimizer. This novel optimizer builds upon the existing Muon algorithm by adding a unique QK-clip technique. This addition addresses training instability, particularly issues like “exploding attention logits,” which can disrupt the learning process in large models. Thanks to MuonClip, Kimi K2 was pre-trained on an enormous 15.5 trillion tokens without a single loss spike, demonstrating remarkable stability and token efficiency.
The pre-training phase also focused heavily on maximizing “token utility” – how much effective learning signal each token contributes. Recognizing the limitations of high-quality human-generated data, the Kimi team developed a sophisticated rephrasing pipeline. This system synthetically generates diverse versions of existing high-quality knowledge and mathematical texts, effectively amplifying the training data without causing overfitting. This approach ensures that Kimi K2 learns more efficiently from the available data.
A Robust Architecture for Agentic Capabilities
Kimi K2’s architecture is a trillion-parameter Mixture-of-Experts transformer. It features 384 experts, with 8 active per token, and utilizes Multi-head Latent Attention (MLA). This design choice was informed by extensive scaling law analysis, which showed that increasing sparsity (the ratio of total experts to active experts) significantly improves performance while maintaining computational efficiency. The model also optimizes for inference efficiency, particularly for long context lengths, by reducing the number of attention heads compared to similar models like DeepSeek-V3.
Multi-Stage Post-Training for Real-World Interaction
The post-training process for Kimi K2 is a sophisticated, multi-stage approach. It begins with large-scale supervised fine-tuning (SFT) using a diverse instruction-tuning dataset. A highlight of this stage is a large-scale agentic data synthesis pipeline. This pipeline simulates real-world tool-use scenarios, generating tens of thousands of high-quality training examples. It involves creating tool specifications, generating diverse agents and tasks, and simulating multi-turn interactions where agents use tools to complete tasks. A crucial quality evaluation and filtering step ensures only successful trajectories are used for training. This synthetic approach is complemented by real execution environments, especially for coding and software engineering tasks, providing authentic feedback.
Following SFT, Kimi K2 undergoes a joint reinforcement learning (RL) stage. This stage is designed to further enhance the model’s capabilities through interactions with real and synthetic environments. The RL framework incorporates verifiable rewards for tasks in math, STEM, logical reasoning, complex instruction following, faithfulness, and coding. Beyond verifiable tasks, Kimi K2 introduces a “Self-Critique Rubric Reward” mechanism, where the model evaluates its own outputs against human preferences like helpfulness and creativity, allowing it to learn from subjective feedback. The RL algorithm also includes features like budget control, an auxiliary PTX loss to prevent forgetting, and a temperature decay schedule to balance exploration and exploitation.
Also Read:
- Innovator: Bridging General and Scientific Knowledge in Large Language Models
- GEPA: A New Approach to Optimizing AI Systems Through Language Reflection
Leading Performance Across Diverse Benchmarks
Kimi K2 has demonstrated state-of-the-art performance across a wide array of benchmarks. In agentic and competitive coding, it achieves leading scores on SWE-bench Verified (65.8%), SWE-bench Multilingual (47.3%), and LiveCodeBench v6 (53.7%), often surpassing both open-source and many closed-source models in non-thinking settings. For multi-turn tool-use tasks, Kimi K2 sets a new standard, scoring 66.1 on Tau2-Bench and 76.5 on ACEBench.
The model also exhibits strong general capabilities, leading open-source peers on MMLU (89.5%), MMLU-Redux (92.7%), and instruction-following benchmarks like IFEval (89.8%). In math and STEM, it achieves top-tier results on AIME 2024 (69.6%) and GPQA-Diamond (75.1%). On the LMSYS Arena leaderboard (as of July 17, 2025), Kimi K2 ranked as the top open-source model and fifth overall based on over 3,000 user votes, highlighting its strong real-world user preference.
While Kimi K2 represents a significant leap forward, the Kimi Team acknowledges certain limitations. These include potential for excessive token generation on hard reasoning tasks or unclear tool definitions, and occasional performance decline if tool use is unnecessarily enabled. The team is actively working to address these in future releases.
The release of Kimi K2’s base and post-trained model checkpoints aims to foster further research and applications in agentic intelligence. For more in-depth technical details, you can refer to the full Kimi K2 Technical Report.
