TLDR: Latent Discrete Diffusion Models (LDDMs) address the limitations of factorized denoisers in discrete diffusion by coupling a masked discrete diffusion over tokens with a continuous diffusion over latent embeddings. This continuous latent channel provides a softer signal and carries cross-token dependencies, improving joint structure and generation quality, especially in few-step scenarios. Two variants, FUJI-LDDMs (fully joint denoising) and SEQ-LDDMs (sequential latent-first denoising), show improved performance on synthetic tasks and unconditional language modeling.
Diffusion models have become a cornerstone in generating high-quality content, especially for continuous data like images and audio. However, applying these powerful techniques to discrete data, such as language, presents unique challenges. Traditional discrete diffusion models, while effective, often face a significant limitation: their reverse transitions typically update individual tokens independently. This ‘factorization bottleneck’ can weaken the joint structure of the generated data and degrade quality, particularly when models are asked to generate content in just a few steps.
A new research paper introduces Latent Discrete Diffusion Models (LDDMs) as a novel approach to overcome this limitation. The core idea behind LDDMs is to combine a masked discrete diffusion process, which handles tokens, with a continuous diffusion process that operates on latent embeddings. This continuous ‘latent channel’ acts as a softer signal, carrying crucial cross-token dependencies that help resolve ambiguities and improve the overall coherence of the generated output.
The researchers propose two main variations of LDDMs. The first, called FUJI-LDDMs (FUlly JoInt denoising), performs a simultaneous denoising of both tokens and their corresponding latent embeddings at each step. This means the discrete and continuous aspects evolve together, allowing for constant interaction and mutual refinement. The second variant, SEQ-LDDMs (SEQuential denoising), takes a different approach. It first resolves the continuous latent chain to its initial state and then uses this resolved latent information to condition the entire discrete token generation process. This sequential method can be particularly effective when a clear, global signal from the latent space can guide the discrete generation.
The paper highlights that the factorized nature of many existing discrete diffusion models leads to a ‘hard commitment’ when unmasking tokens. Unlike continuous diffusion, where errors can be amortized through small, reversible adjustments, unveiling a token in a discrete model is a final decision. LDDMs address this rigidity by providing a continuous, auxiliary signal that helps stabilize training and improve the consistency of outputs, especially when many tokens are unmasked at once.
In their experiments, LDDMs demonstrated notable improvements. On a synthetic task designed to test conditional factorization, SEQ-LDDMs achieved near-optimal results with very few data steps, showcasing its ability to leverage the latent channel effectively. For unconditional language modeling on a large text corpus (LM1B), FUJI-LDDMs yielded lower generative perplexity across various sampling budgets, indicating better text quality, while maintaining similar levels of token entropy compared to state-of-the-art masked discrete diffusion baselines. These gains were most pronounced when the models were allowed fewer sampling steps, a desirable characteristic for faster generation.
The researchers also discuss practical aspects of training LDDMs, including objective functions derived from the Evidence Lower Bound (ELBO) and design choices for learning informative latents. They found that normalizing encoder outputs to unit norm was a simple yet effective fix to prevent latent ‘blow-up’ and stabilize training. For the language modeling task, they even utilized a frozen, pre-trained Qwen3 sentence encoder to provide the initial latent representations, demonstrating that LDDMs can effectively integrate existing continuous representations.
Also Read:
- Planned Diffusion: A Hybrid Method for Efficient LLM Text Generation
- Efficient One-Step Generation with Di-Bregman Diffusion Distillation
In conclusion, Latent Discrete Diffusion Models offer a promising direction for improving generative performance on discrete data. By intelligently coupling discrete token generation with a continuous latent space, LDDMs provide a mechanism to capture and leverage joint structure, leading to more coherent and higher-quality outputs, particularly in scenarios requiring efficient, few-step generation. You can read the full research paper here.
