TLDR: Direct Multi-Token Decoding (DMTD) is a novel method to accelerate large language model (LLM) inference. It capitalizes on the functional specialization of LLM layers, allowing the model to generate multiple tokens in a single cycle by reusing only the late layers after an initial full pass. This approach avoids additional parameters or post-generation verification, achieving up to a 2x speedup with minimal performance degradation, particularly benefiting larger models and scaling well with increased training data.
Large Language Models (LLMs) have become the cornerstone of modern AI, powering everything from chatbots to content generation. These models, often built on decoder-only transformer architectures, are incredibly powerful but can be computationally intensive, especially during the inference (generation) phase. A new research paper introduces an innovative approach called Direct Multi-Token Decoding (DMTD) that promises to significantly speed up LLM inference without adding complexity or requiring auxiliary models.
The core idea behind DMTD stems from recent observations about how LLM layers specialize. Researchers have found that early layers primarily focus on understanding the input context, middle layers handle task-specific processing and reasoning, and late layers are responsible for converting abstract representations into actual output tokens. Traditionally, when an LLM generates text, it performs a full pass through all these layers for each token it produces, even if the early and middle layers might not need to be re-evaluated for every single token.
The authors of the paper, Xuan Luo, Weizhi Wang, and Xifeng Yan from UC Santa Barbara, hypothesized that once the input has been processed by the early and middle layers, the resulting internal states might contain enough information for the late layers to generate multiple tokens consecutively. This would eliminate the need to repeatedly traverse the computationally expensive early and middle layers for every subsequent token.
How Direct Multi-Token Decoding Works
DMTD operates in fixed ‘multi-token cycles’. Instead of generating tokens one by one through full forward passes, DMTD performs one full forward pass at the beginning of a cycle. After this initial pass, it reuses only the late layers to decode multiple tokens consecutively within that same cycle. This approach is remarkably minimal: it introduces no additional parameters, auxiliary routines, or post-generation verification steps, which are often required by other acceleration methods like speculative decoding.
To enable this, DMTD employs a ‘cyclical masking strategy’ during training, allowing the model to learn to predict multiple future tokens simultaneously from a single input sequence. For inference, a ‘cyclical refilling mechanism’ is introduced. This mechanism ensures that the necessary intermediate information (known as KV cache entries) for the early and middle layers is restored from previous cycles, maintaining a complete context for generation even when these layers are skipped.
Also Read:
- Dynamic Nested Depth: A New Approach to Smarter Language Models
- CacheClip: Boosting RAG System Speed and Accuracy with Smart KV Cache Reuse
Performance and Efficiency Gains
The speedup achieved by DMTD is particularly significant because LLM inference is often ‘memory-bound’ rather than ‘compute-bound’. This means that the speed is more dependent on the number of layers processed rather than the total computational volume. By processing fewer layers per token, DMTD capitalizes on this characteristic to accelerate inference.
Experiments conducted on a fine-tuned Qwen3-4B model, reusing the last 8 of its 36 layers as decoding layers, demonstrated promising results. With a cycle length of 4 (generating 4 tokens per cycle), DMTD achieved up to a 2x speedup in inference time with only a minor performance loss, retaining 96.3% of the vanilla model’s overall performance. The method showed even better performance retention (100%) when decoding two tokens per cycle.
The research also highlighted that DMTD’s performance improves with larger training datasets and is particularly well-suited for larger language models. For instance, the Qwen3-4B model retained 98.4% of its original performance with DMTD, compared to 72.6% for the smaller Qwen3-0.6B model. This suggests that bigger models, with their increased parameters and dimensionality, are better equipped to encode the ‘anticipatory information’ needed for multi-token prediction.
While the current study used a limited dataset for fine-tuning, the scaling analysis indicates that DMTD’s performance is expected to further improve with larger-scale continued pre-training. The flexibility of DMTD also allows a single model to dynamically adjust the inference cycle length, balancing speedup and quality as needed.
Direct Multi-Token Decoding represents a significant step forward in making LLM inference more efficient. By intelligently reusing existing model layers, it offers a simple yet powerful way to accelerate text generation without compromising model integrity. For more technical details, you can read the full research paper here.
