TLDR: This research paper introduces Coevolutionary Continuous Discrete Diffusion (CCDD), a novel language modeling paradigm that combines the strengths of continuous and discrete diffusion models. It addresses the limitations of existing LLMs and diffusion models by defining a joint multimodal diffusion process on both continuous representation space and discrete token space. CCDD leverages the theoretical expressivity of continuous diffusion and the practical trainability of discrete diffusion, enhanced by techniques like contextualized embeddings and asynchronous noise schedules. Experimental results show significant improvements in language modeling performance, demonstrating CCDD’s ability to achieve rich semantics and high sample quality.
Recent advancements in large language models (LLMs) have showcased impressive capabilities, particularly in complex reasoning tasks. However, these models, often based on autoregressive transformers, inherently operate in a discrete, left-to-right manner. This fundamental design choice can limit their expressivity, making certain problems, such as recognizing formal languages or understanding graph connectivity, challenging to address accurately. Even state-of-the-art LLMs can struggle with tasks requiring extensive planning, parallel searching, and backtracking, like solving Sudoku.
Researchers have explored new paradigms to overcome these limitations. One approach involves enhancing LLMs with ‘latent reasoning’ through techniques like looped transformers (LTs) or continuous Chain-of-Thought (CoT). Looped transformers, for instance, theoretically demonstrate improved expressivity by repeatedly applying a block of transformer layers, allowing for complex reasoning in a continuous latent space before decoding into discrete tokens. Despite their theoretical power, LTs haven’t seen widespread adoption due to practical performance issues, largely attributed to a lack of supervision on their intermediate states during training.
Another promising direction is Diffusion Language Models (DLMs). These models offer a non-autoregressive approach to text generation, enabling capabilities like any-order generation, self-correction, and parallel decoding. DLMs have shown superior performance in structured reasoning tasks. They generally fall into two categories: Continuous Diffusion Models (CDMs), which operate in embedding or probability spaces, and Discrete Diffusion Models (DDMs), which work with discrete tokens. Intriguingly, while CDMs are theoretically more expressive than DDMs and LTs, they often underperform their discrete counterparts in practice.
This discrepancy between theoretical expressiveness and empirical performance is a key challenge. The authors of the paper, “Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner,” argue that this contradiction stems from practical trainability issues. While CDMs offer intermediate supervision that LTs lack, they face difficulties in decoding continuous representations back into discrete tokens. Discrete diffusion, on the other hand, operates in the explicit token space, which can lead to information loss during decoding steps.
To bridge this gap, a new language modeling paradigm called Coevolutionary Continuous Discrete Diffusion (CCDD) has been proposed. CCDD defines a joint multimodal diffusion process that operates simultaneously on a continuous representation space and a discrete token space. This innovative approach leverages a single model to denoise in both spaces concurrently. By combining these two modalities, CCDD aims to achieve the best of both worlds: the rich semantics and expressivity of continuous diffusion in the latent space, coupled with the good trainability and sample quality provided by explicit discrete tokens.
The core idea behind CCDD is to allow the continuous space to preserve full semantics from previous denoising steps, which can then be used as historical information. Simultaneously, the discrete component provides a more confident, radical decoding mechanism. The model uses a single time-conditioned network that takes inputs from both discrete and continuous modalities and outputs modality-specific predictions. The training loss for CCDD is a weighted sum of losses from both continuous and discrete components.
The implementation of CCDD incorporates effective architectural designs, drawing inspiration from Diffusion Transformers (DiT) and Mixture-of-Experts (MoE) models. A crucial aspect is the selection of the continuous space, where contextualized embeddings from well-pretrained LLMs, such as Qwen3-Embedding, are utilized. These high-quality embeddings provide rich sequence-level semantics and serve as a smoother target for generation, while the discrete diffusion aids in the decoding process. This can also be seen as a form of knowledge distillation from pretrained LLMs.
Further enhancements include classifier-free guidance (CFG), which treats continuous representations as self-generated guidance for discrete token generation, and asynchronous noise schedules. The asynchronous schedules are designed to allow latent representations to be generated faster in the continuous space, acting as high-level guidance for token decoding.
Experimental results on datasets like One Billion Words (LM1B) and OpenWebText (OWT) demonstrate the effectiveness of CCDD. The model significantly reduces validation perplexity compared to discrete-only baselines, even when using the same number of parameters. For instance, CCDD reduced validation perplexity by over 25% compared to the MDLM baseline on the LM1B dataset. This strong empirical performance validates the benefits of joint modeling and the synergy between continuous and discrete diffusion components.
Also Read:
- Google Unveils TUMIX: A Novel Multi-Agent Framework for Enhanced LLM Performance and Cost Efficiency
- HKU Business School Report Reveals Chinese AI Models Lag in Hallucination Control
In conclusion, CCDD represents a novel step in language modeling, offering a framework that combines the theoretical expressivity of continuous diffusion with the practical trainability of discrete diffusion. By addressing the limitations of previous models and leveraging the strengths of multimodal spaces, CCDD paves the way for more powerful and efficient language models. For more details, you can read the full research paper here.
