TLDR: Dynamic Nested Depth (DND) is a novel method that enhances Large Language Models (LLMs) by adaptively identifying critical tokens and reprocessing them in a nested manner. This “review” process allocates extra computation to difficult tokens while avoiding redundant work for easier ones. DND integrates into existing dense and Mixture-of-Experts (MoE) models post-training, achieving significant performance improvements across diverse tasks (language, math, coding) with minimal increases in parameters and computation. Its effectiveness is ensured by tailored training strategies that control token selection precision and stability.
Large Language Models (LLMs) have revolutionized artificial intelligence, but their continuous improvement often relies on scaling up model size, data, and computation. This approach, however, comes with significant computational overhead during both training and inference. A new method, Dynamic Nested Depth (DND), offers an efficient way to enhance LLM performance without simply making models larger.
Addressing Computational Inefficiency
The core idea behind DND stems from the observation that not all parts of an input sequence are equally challenging for an LLM to process. Some tokens are straightforward, while others require deeper computational effort. Traditional methods like token pruning reduce computation by discarding less important tokens. DND takes this a step further by allocating additional computation to the most critical tokens, ensuring they are thoroughly understood.
How Dynamic Nested Depth (DND) Works
DND is designed to be integrated directly into pre-trained LLMs during a post-training phase. It operates by dynamically selecting a subset of tokens that are deemed more difficult and reprocessing them through the transformer layers. This process is akin to the model “reviewing” challenging parts of the input.
The DND architecture involves three main components:
- Token Selection (Router): At the end of a transformer layer, a specialized linear layer acts as a router. It evaluates each token’s hidden state and assigns a preference score, which is then converted into a probability. If this probability exceeds a predefined threshold, the token is marked for additional processing. This “token-choice” routing ensures independent decisions for each token, suitable for auto-regressive models.
- Nested Depth Processing: Once selected, these critical tokens are gathered into a new, compact sequence. They are then fed back through the same transformer layer, but with new positional embeddings. This “nested pass” allows the model to refine the representations of these important tokens through an internal iteration.
- Normalized Fusion: To preserve the global knowledge learned during pre-training, the output from the original “vanilla pass” and the “dynamic nested pass” are merged. A learnable parameter and the token’s routing score modulate this fusion, giving more weight to the recomputed representation for tokens identified as more difficult.
Smart Training for Stable Performance
To ensure DND’s effectiveness, a carefully designed training strategy is employed. This strategy focuses on two key aspects: making the router’s token selection precise and keeping the selection stable.
- Router Controlling Loss: This loss function encourages the router to produce scores that are clearly distinguishable, preventing them from clustering together. It also ensures the router remains sensitive to its inputs by keeping scores within the dynamic range of the sigmoid function, avoiding issues like vanishing gradients.
- Threshold Control Scheme: A dynamic mechanism adjusts the selection threshold in real-time. This “buffer proportional control” corrects the threshold based on the difference between the actual and desired token selection ratio. Additionally, “EMA synchronization” periodically aligns the threshold with a smoothed ideal value, promoting long-term training stability.
Also Read:
- EAGER: Smarter LLM Reasoning Through Adaptive Compute Allocation
- Adapting Knowledge Distillation for Efficient Large Language Models
Demonstrated Impact and Efficiency
The effectiveness of DND has been demonstrated by integrating it into both dense and Mixture-of-Experts (MoE) LLMs. Experiments showed significant performance boosts across various benchmarks, including language, mathematics, reasoning, and coding tasks. For instance, the dense Qwen3-1.7B model saw a 1.88% performance increase, and the MoE Qwen3-30B-A3B model improved by 0.87% on average. These gains were achieved with a minimal increase in parameters (less than 0.1M) and computing, highlighting DND as a highly efficient approach.
Ablation studies confirmed that both the DND architecture and its specialized training strategies are crucial for these performance improvements. Analysis of token selection revealed that DND learns to prioritize different types of tokens at different layers, with shallower layers focusing on essential nouns and deeper layers on more abstract or syntactically critical elements like mathematical expressions or key verbs.
In conclusion, Dynamic Nested Depth (DND) presents a powerful and efficient paradigm for enhancing Large Language Models. By adaptively identifying and selectively deepening the computation for critical tokens, DND unlocks significant capabilities in LLMs with minimal additional resources. You can read the full research paper for more technical details here: DND: Boosting Large Language Models with Dynamic Nested Depth.
