TLDR: The paper introduces Authentic Discrete Diffusion (ADD), a novel framework that redefines discrete generative modeling by operating directly in the one-hot space. Unlike prior methods that rely on continuous latent spaces or masking, ADD uses Gaussian-corrupted one-hot vectors and iteratively denoises them with a timestep-conditioned cross-entropy loss and arg max-based discretization. It demonstrates superior performance in image classification and text generation (image captioning) tasks, offering a stable and effective approach for true discrete-space diffusion.
Diffusion models have emerged as a powerful tool in generative artificial intelligence, particularly excelling in tasks involving continuous data like image synthesis. These models work by gradually corrupting data with Gaussian noise and then learning to reverse this process, effectively denoising the data back to its original form. However, applying these models to discrete data, which has a categorical structure and non-Euclidean geometry, has always presented significant challenges.
Traditional approaches to discrete diffusion have generally fallen into two categories, both with limitations. One method involves mapping discrete variables into continuous latent spaces, but this often leads to degraded generation quality and unstable training. Another common approach, often referred to as “pseudo” discrete diffusion (PDD), frames the problem as a masked modeling task, similar to how BERT works. Here, noise is simulated by randomly masking tokens, and the model is trained to reconstruct the original input. While these methods can be effective in some applications, they don’t truly adhere to the formal definition of a diffusion process and often perform less effectively than strong autoregressive models.
Introducing Authentic Discrete Diffusion (ADD)
A new framework, the Authentic Discrete Diffusion (ADD) model, aims to fundamentally redefine how diffusion models handle discrete data. Unlike its predecessors, ADD preserves the core characteristics of diffusion directly within the one-hot space. This means it works directly with categorical data, such as one-hot encoded labels, without needing to convert them into continuous latent spaces or rely on masking policies.
At its core, ADD starts with one-hot vectors that have been corrupted by Gaussian noise. It then iteratively denoises these vectors over multiple steps, following a carefully defined noise schedule. During the inference process, the model predicts a clean one-hot vector by first applying an arg max operation to its output probabilities, which then converts the result into a one-hot representation. This refined vector is then fed back into the next iteration after adding noise with a reduced coefficient. This autoregressive-style feedback loop allows for progressive refinement, gradually eliminating uncertainty and converging towards precise, categorical outputs.
A crucial innovation in ADD is the introduction of a timestep-conditioned cross-entropy loss. This loss function directly enforces a correspondence between the model’s predicted one-hot vectors and the original target labels. This design helps prevent the diffusion network from over-relying on conditioning signals and ensures that the outputs respect the mutually exclusive nature of discrete categories, avoiding the smoothing effects often seen with mean squared error (MSE) losses used in continuous diffusion models.
Applications and Performance
The researchers evaluated ADD on two key tasks: large-scale image classification using the ImageNet benchmark and image-conditioned text generation (image captioning) using the COCO Captions dataset. The results are compelling. For classification, ADD not only achieved superior performance compared to traditional classifier baselines but also surpassed larger models with significantly more parameters. For instance, the “ViT-Base (ADD)” model achieved a top-1 accuracy of 82.8%, outperforming “ViT-Large (standard classification)” which uses nearly three times more parameters.
In text generation, ADD demonstrated excellent capabilities. When compared to pseudo-discrete diffusion (PDD) models, ADD-generated captions exhibited strong semantic alignment with images, as measured by CLIP Scores, and were qualitatively more fluent and grammatically correct. PDD, in contrast, often produced ungrammatical or semantically inconsistent text. This highlights ADD’s ability to model discrete sequential dependencies effectively.
Also Read:
- GLASS Flows: A New Approach to Efficient Stochastic Sampling in Generative AI
- SiD-DiT: Bridging Diffusion and Flow Matching for Faster Image Synthesis
Key Contributions and Future Directions
The Authentic Discrete Diffusion framework represents a significant step forward in generative modeling for categorical domains. Its key contributions include operating entirely in the one-hot space, introducing a timestep-conditioned cross-entropy loss for direct categorical supervision, and designing an iterative refinement mechanism that progressively denoises Gaussian-corrupted one-hot vectors. The extensive ablation studies confirmed the indispensability of each component, particularly the cross-entropy alignment loss and classifier-free guidance, in ensuring the stability and effectiveness of discrete diffusion.
This work provides a robust foundation for future research in discrete generative modeling, with potential applications in large-scale language tasks, multimodal reasoning, and structured prediction problems. For more in-depth technical details, you can read the full research paper here.
