TLDR: This research introduces GRAFT and P-GRAFT, novel frameworks for fine-tuning diffusion models by unifying rejection sampling and implicitly performing PPO with reshaped rewards, overcoming the intractability of marginal likelihoods. P-GRAFT, which shapes distributions at intermediate noise levels, empirically leads to more effective fine-tuning across text-to-image, layout, and molecule generation. Additionally, Inverse Noise Correction is proposed for flow models, improving generation quality and efficiency without explicit rewards, demonstrated in unconditional image generation.
Diffusion models have become incredibly powerful tools for generating all sorts of content, from stunning images to complex molecular structures. These models learn to create new data by gradually removing noise from a random starting point. While pre-trained diffusion models are good at capturing the general characteristics of their training data, there’s often a need to fine-tune them for specific tasks or to align them with particular preferences, often guided by a ‘reward function’.
Traditionally, methods like Proximal Policy Optimization (PPO), a type of reinforcement learning, have been very successful in fine-tuning other generative models, especially in language. However, applying PPO directly to diffusion models is tricky because it requires calculating ‘marginal likelihoods’ – a measure of how probable a generated sample is – which are notoriously difficult to compute for diffusion models. This challenge has led researchers to look for alternative approaches.
A new research paper, titled Fine-Tuning Diffusion Models via Intermediate Distribution Shaping, introduces a novel framework called GRAFT (Generalized Rejection sAmpling Fine-Tuning) that offers an elegant solution. GRAFT unifies various rejection sampling techniques, which are methods that select high-quality samples based on their rewards. The paper demonstrates that GRAFT implicitly performs PPO, but with a cleverly ‘reshaped’ reward function. This allows diffusion models to benefit from PPO-like optimization without needing to tackle the intractable marginal likelihood problem.
Building on GRAFT, the researchers propose an even more effective method called P-GRAFT (Partial-GRAFT). Instead of fine-tuning the model based on the final, fully denoised outputs, P-GRAFT focuses on shaping the distribution of samples at ‘intermediate noise levels’. Imagine a sculptor refining a clay model: P-GRAFT suggests that making adjustments when the model is still a bit rough (noisy) can be more impactful than waiting until the very end. The paper provides a mathematical explanation for this, highlighting a ‘bias-variance tradeoff’. Essentially, while rewards might be noisier at intermediate stages, the learning problem itself becomes simpler, leading to more effective fine-tuning.
The benefits of P-GRAFT were demonstrated across a range of applications. In text-to-image generation, applying P-GRAFT to Stable Diffusion 2 significantly improved image quality and alignment with text prompts, outperforming existing policy gradient methods. It also showed impressive gains in layout generation and molecule generation, where the goal is to create stable molecular structures. Notably, P-GRAFT often achieved better results with fewer training samples compared to other methods.
Also Read:
- MaskGRPO: A Unified Reinforcement Learning Approach for Multimodal Discrete Diffusion Models
- DMIS: A Unified Framework for Robust Diffusion Models with Imperfect Supervision
Beyond Rewards: Inverse Noise Correction for Flow Models
The paper also introduces an innovative technique called Inverse Noise Correction, specifically for ‘flow models’. Unlike diffusion models that use stochastic processes, flow models use deterministic paths to transform noise into data. Inverse Noise Correction aims to improve the quality of flow models even without relying on explicit reward functions. It works by first training a flow model, then using it in reverse to generate an ‘inverse noise’ distribution from real data. A second ‘noise corrector’ model is then trained to transform standard random noise into this learned inverse noise. During inference, the corrected noise is fed into the original flow model to generate higher-quality images.
This method proved highly effective for unconditional image generation on datasets like CelebA-HQ and LSUN-Church. It not only improved the quality of generated images (measured by FID scores) but also allowed for faster generation, achieving better results with fewer computational steps. This is a significant advantage, as it means higher quality can be achieved with less computational effort.
In summary, this research presents a comprehensive framework for fine-tuning generative models, particularly diffusion models, by leveraging rejection sampling and intermediate distribution shaping. The introduction of GRAFT and P-GRAFT provides powerful tools for aligning these models with specific objectives, while Inverse Noise Correction offers a novel, reward-free approach to enhance flow model performance. These advancements pave the way for more controllable and efficient generative AI applications.
