TLDR: NVIDIA has released Nemotron Nano 2, a new hybrid Mamba-Transformer language model designed for high-throughput reasoning. Pre-trained on 20 trillion tokens and compressed from 12 billion to 9 billion parameters, it offers up to 6x higher inference throughput than comparable models like Qwen3-8B, while maintaining or improving accuracy. The model is optimized for long-context inference (up to 128k tokens) on a single NVIDIA A10G GPU and is open-sourced on Hugging Face along with its training datasets.
NVIDIA has unveiled its latest advancement in large language models, the Nemotron Nano 2. This new model is a hybrid Mamba-Transformer architecture designed to significantly boost the speed of reasoning tasks while maintaining top-tier accuracy. It builds upon the foundation of the Nemotron-H architecture, replacing most self-attention layers with Mamba-2 layers to enhance inference speed, especially for the long ‘thinking traces’ required in complex reasoning.
The journey to Nemotron Nano 2 began with the pre-training of a larger 12-billion-parameter model, Nemotron-Nano-12B-v2-Base. This extensive training involved 20 trillion tokens and utilized an FP8 training method. Following this, the model underwent an alignment process and was then compressed and refined using a strategy called Minitron. This compression aims to enable the model to perform inference on up to 128,000 tokens using a single NVIDIA A10G GPU, which has 22 GiB of memory and uses bfloat16 precision.
Compared to other models of similar size, such as Qwen3-8B, Nemotron Nano 2 demonstrates comparable or superior accuracy on various reasoning benchmarks. More impressively, it achieves up to six times higher inference throughput in scenarios demanding significant output, like processing 8,000 input tokens and generating 16,000 output tokens. This efficiency makes it particularly well-suited for generation-heavy applications.
NVIDIA is making several versions of the model available on Hugging Face, including the aligned and pruned Nemotron-Nano-9B-v2, a pruned base model called Nemotron-Nano-9B-v2-Base, and the original Nemotron-Nano-12B-v2-Base before its alignment and pruning. Alongside these models, the company is also releasing a substantial portion of its pre-training and post-training datasets. The pre-training dataset, Nemotron-Pre-Training-Dataset-v1, contains over 6 trillion tokens and includes diverse categories like web crawl data (Nemotron-CC-v2), a 133-billion-token math dataset (Nemotron-CC-Math-v1), curated GitHub code references (Nemotron-Pretraining-Code-v1), and synthetic SFT-style data (Nemotron-Pretraining-SFT-v1). An updated post-training dataset, Nemotron-Post-Training-Dataset-v2, will also be released, expanding SFT and reinforcement learning data into five target languages to improve math, code, general reasoning, and instruction-following capabilities.
The architecture of Nemotron-Nano-12B-v2-Base is a blend of Mamba-2, self-attention, and Feed-Forward Network (FFN) layers. It features 62 layers in total, with a specific distribution of these different layer types. The model was pre-trained on a vast and high-quality corpus of both curated and synthetically generated data, including English and multilingual web crawl data, specialized math data, and extensive code data. Synthetic data generation played a crucial role, covering STEM subjects, additional math problems, multilingual Q&A, code Q&A, and academic content, as well as various SFT-style datasets focused on fundamental reasoning.
The pre-training process involved a curriculum-based approach with three phases of data blending, prioritizing diversity initially and then shifting towards higher-quality datasets. A long-context extension phase was also incorporated to ensure the model’s capability to infer over extended context windows, specifically targeting 128,000 tokens. This involved continuous pre-training with very long sequence lengths and the generation of high-quality long-context synthetic data.
The alignment process for Nemotron Nano 2 involved multiple stages of supervised fine-tuning (SFT), Group Relative Policy Optimization (GRPO), Direct Preference Optimization (DPO), and Reinforcement Learning from Human Feedback (RLHF). These stages refined the model’s ability in areas like instruction adherence, tool-calling, and conversational skills. A unique feature is the model’s budget control, allowing users to specify the number of ‘thinking tokens’ the model can generate before producing a final answer, which was improved through specific truncated training examples.
Also Read:
- QuickMerge++: Streamlining Generative AI with Adaptive Token Compression
- Uncertainty and Importance: New Keys to Efficient Wireless LLM Inference
For compression, the 12B model was reduced to 9B parameters through pruning of full layers, FFN hidden size, and embedding channels. This was crucial to enable efficient inference on the NVIDIA A10G GPU. A lightweight Neural Architecture Search framework was used to identify the optimal architectural candidates that met memory and throughput objectives. The final Nemotron-Nano-9B-v2 model retains 56 layers and has reduced embedding channels and FFN intermediate sizes, achieving its impressive performance and efficiency. For more technical details, you can refer to the full research paper.
