TLDR: A new framework called Dosser improves privacy-preserving dataset distillation by decoupling data sampling from optimization and using subspace projection to reduce noise in training signals. This leads to better accuracy and efficiency in creating compact, private synthetic datasets, especially on complex image datasets like CIFAR-10, by more effectively utilizing the privacy budget and enhancing signal quality.
In the rapidly evolving landscape of machine learning, the demand for vast datasets to train powerful and accurate models is ever-increasing. However, these datasets frequently contain sensitive and private information, leading to significant privacy concerns. Addressing this, Differential Privacy (DP) offers a robust solution by generating synthetic datasets that limit the leakage of private information within a defined privacy budget. Yet, a common challenge with DP data generation is the need for substantial amounts of synthetic data to achieve performance comparable to models trained on original data, leading to high storage and computational costs.
Dataset Distillation (DD) emerges as a promising alternative, renowned for its efficiency in training and storage. DD aims to condense large datasets into smaller, highly informative synthetic sets, allowing models trained on these distilled sets to perform similarly to those trained on much larger original datasets. While DD excels at creating compact and visually anonymized data, it doesn’t inherently provide privacy guarantees.
The integration of Dataset Distillation with Differential Privacy is a critical area of research, aiming to combine the compactness of DD with the stringent privacy guarantees of DP. However, existing methods in this domain face significant limitations. Many current private DD techniques suffer from a synchronized sampling-optimization process, meaning that every step of refining the synthetic data requires a new, noisy query from the private dataset. This leads to an inefficient use of private information due to the accumulation of excessive noise. Furthermore, these methods often rely on randomly initialized neural networks to extract training signals, which tend to capture uninformative details, resulting in a low signal-to-noise ratio (SNR) and amplifying the negative impact of added DP noise.
Introducing Dosser: A Novel Framework for Enhanced Privacy-Preserving Dataset Distillation
To overcome these challenges, researchers have introduced a novel framework called Dosser, which stands for Decoupled Optimization and Sampling with Subspace-based Error Reduction. This innovative approach aims to maximize the utility of training signals from two key perspectives, leading to more efficient and accurate privacy-preserving dataset distillation.
Decoupled Optimization and Sampling (DOS)
The first core innovation in Dosser is the decoupling of the sampling process from the optimization process. In traditional methods, these two stages are intertwined, forcing a large number of noisy sampling steps if many optimization iterations are needed for convergence. Dosser separates these, allowing for a fixed number of private training signals to be sampled under a DP budget in an initial ‘sampling stage’. Once these noisy, aggregated signals are collected, they are stored and then used repeatedly in a separate ‘optimization stage’ to refine the synthetic dataset over a much larger number of iterations. This decoupling means that extended optimization can occur without incurring additional privacy costs, as no new noise is added during the optimization phase, leading to better convergence and improved image quality.
Subspace-based Error Reduction (SER)
The second key innovation is Subspace-based Error Reduction (SER), which focuses on improving the signal-to-noise ratio (SNR) of the raw extracted signals. SER leverages auxiliary datasets to identify an ‘informative subspace’ within the randomly initialized neural networks. By projecting the training signals into this learned subspace, Dosser effectively filters out uninformative noise components that are typically captured by random networks. This concentration of signal power on high-utility dimensions significantly enhances the SNR, thereby mitigating the impact of the added DP noise. The auxiliary dataset can be generated either using pre-trained foundational models (like Stable Diffusion for natural images) or by training a differentially private generative model on the private dataset itself, ensuring no additional privacy leakage.
The Synergy of DOS and SER
Together, DOS and SER create a powerful synergy. DOS provides the flexibility for extensive optimization, while SER ensures that the signals being optimized are of higher quality and less corrupted by noise. This combined approach allows Dosser to better balance privacy and utility, enabling the creation of compact yet highly informative synthetic datasets.
Also Read:
- New Strategies for Handling Client Dropouts in Decentralized Federated Learning
- Unmasking Malicious Clients in Federated Learning with Watermarks
Experimental Validation and Impact
The effectiveness of Dosser has been rigorously evaluated on standard datasets such as MNIST, FashionMNIST, and CIFAR-10. Under a strict privacy budget, Dosser demonstrated significant accuracy improvements compared to previous state-of-the-art differentially private dataset distillation methods. For instance, on CIFAR-10, Dosser achieved a 10.0% improvement with 50 images per class and an 8.3% increase with just one-fifth the distilled set size of prior methods. It also notably narrowed the accuracy gap between private and non-private dataset distillation, showcasing its superior ability to mitigate noise effects within the differential privacy framework.
While Dosser marks a substantial advancement, the authors acknowledge certain limitations. The current framework is primarily designed for matching training signals from randomly initialized networks, and adapting it to more advanced DD techniques that utilize pre-trained models or trajectory matching remains a future research direction. Additionally, SER’s performance relies on the auxiliary dataset closely matching the distribution of the training data, which might be challenging for highly specialized domains with limited data.
In conclusion, Dosser sets a new standard for privacy-preserving data synthesis by offering a robust framework that enhances noise efficiency through decoupled optimization and intelligent subspace projection. This work represents a significant step forward in making machine learning models more private without sacrificing efficiency or accuracy. For more details, you can refer to the full research paper: Improving Noise Efficiency in Privacy-preserving Dataset Distillation.
