TLDR: Researchers introduce Information-Guided Diffusion Sampling (IGDS), a novel method for dataset distillation that improves performance, especially in low images-per-class (IPC) settings. By focusing on preserving both ‘prototype’ (label-relevant) and ‘contextual’ (intra-class variability) information using a new information-theoretic approach and variational estimators, IGDS generates more diverse and informative distilled datasets, outperforming existing diffusion-based methods.
In the world of deep learning, the success of powerful neural networks often relies on vast, highly informative datasets. However, the sheer size of these datasets can be a significant burden on storage and computational resources during model training. This is where dataset distillation comes in, a technique designed to create a much smaller, compact dataset that still retains the essential information of the original, allowing models trained on it to perform well.
While various methods exist for dataset distillation, recent advancements have seen generative models, particularly diffusion models (DMs), emerge as a strong contender. These models can synthesize new, compact datasets efficiently and in a way that is less dependent on the specific architecture of the deep neural network being trained. Despite their promise, diffusion models face a notable challenge: they struggle in low images-per-class (IPC) settings. In such scenarios, where there are very few examples for each category, the generated samples often lack diversity, leading to a distilled dataset that loses crucial information and performs poorly.
A new research paper, titled “Information-Guided Diffusion Sampling for Dataset Distillation,” addresses this limitation by introducing a novel approach called Information-Guided Diffusion Sampling (IGDS). The core idea behind IGDS is to tackle the problem from an information-theoretic perspective, identifying two key types of information that a distilled dataset must preserve:
Prototype Information (I(X; Y))
This captures the label-relevant features, essentially how much information the input data provides about its correct category. It ensures that the distilled dataset remains effective for classification tasks.
Also Read:
- Unlocking Data Groupings with Diffusion Models: Introducing CLUDI
- Unlocking Image Generation Potential with Cloud Diffusion Models
Contextual Information (H(X|Y))
This preserves the variability and richness within each class, capturing the remaining information in the data once its label is known. It prevents the distilled dataset from collapsing into a narrow set of features, thus maintaining diversity and nuance.
The researchers observed that the required amount of contextual information scales with the IPC. Higher IPC settings benefit from more contextual detail, while lower IPC settings need a stronger focus on prototype information. Building on this, IGDS proposes maximizing a combination of these two information types during the diffusion model’s sampling process, with a dynamic weighting factor that adjusts based on the IPC.
Since directly calculating these information quantities is complex, IGDS develops “variational estimations” using a data-driven approach. This involves training a separate deep neural network, referred to as a variational estimator (VE), which provides tight lower bounds on both prototype and contextual information. Once trained, this VE is integrated into the diffusion model’s sampling process, guiding the generation of distilled data to maximally preserve both types of information.
Experiments conducted on benchmark datasets like Tiny ImageNet and subsets of ImageNet (ImageNette and ImageWoof) demonstrate that IGDS significantly outperforms existing methods, particularly in low-IPC scenarios where prior diffusion-based approaches fall short. For instance, at very low IPC settings, other generative methods often perform similarly to simply selecting a random subset of the original data, whereas IGDS shows a marked improvement.
To illustrate the semantic meaning of these information types, the paper visualizes samples generated by IGDS. When only prototype information is maximized (low contextual information), the synthetic images often feature plain backgrounds and minimal context. As more contextual information is incorporated, the generated images become richer and more diverse, reflecting real-world variations. The method also shows robust performance across different model architectures and achieves strong results on the CIFAR-10 dataset.
For a deeper dive into the methodology and results, you can access the full research paper here: Information-Guided Diffusion Sampling for Dataset Distillation.
