TLDR: PROPS is a novel multi-stage framework for aligning Large Language Models (LLMs) that significantly enhances the privacy of human preference data while improving model utility. It addresses limitations of existing privacy methods like DP-SGD and Randomized Response by progressively refining private labels using intermediate models and Maximum Likelihood Estimation. Experiments show PROPS achieves up to 3x higher win-rates compared to other methods, especially in high-privacy settings, offering a better balance between privacy preservation and alignment quality.
Large Language Models, or LLMs, have become incredibly powerful tools, capable of generating human-like text, answering questions, and even assisting in complex tasks. A crucial step in developing these models is ‘alignment,’ which ensures they adhere to human values and societal norms. This process often relies on human feedback, where people provide preferences on how an LLM should respond to different prompts. However, this reliance on human feedback introduces a significant privacy concern: how much do a labeler’s preferences reveal about their personal values, beliefs, or even identity?
Existing methods for privacy-preserving LLM alignment, such as Differentially Private SGD (DP-SGD) and Randomized Response (RR), offer privacy guarantees but often come at a cost. DP-SGD, for instance, protects the entire training data, including prompts and responses, by adding noise to gradients during training. While robust, this can significantly reduce the model’s utility, especially under strict privacy requirements. Randomized Response, which simply flips preference labels with a certain probability, is simpler but also introduces substantial noise, degrading alignment quality.
A new research paper introduces an innovative framework called PROPS (PROgressively Private Self-alignment) that aims to address these limitations. PROPS focuses specifically on ‘preference-level privacy,’ meaning it protects the privacy of the human-provided preference labels, recognizing that these are the most sensitive part of the data. The core idea behind PROPS is a multi-stage alignment process where models aligned in earlier stages can help label data for subsequent stages, progressively refining the alignment while maintaining privacy.
How PROPS Works: A Staged Approach
Imagine the alignment process as a journey with multiple stops. PROPS divides the human preference dataset into disjoint subsets. In the first stage, a portion of the dataset has its human-provided labels perturbed using Randomized Response. This ‘noisy’ data is then used to train an initial, privately aligned LLM, let’s call it M1.
In the second stage, the magic happens. Instead of relying solely on more noisy human labels, PROPS leverages the intermediate model M1. M1 is used to generate its own rankings for the remaining, unperturbed prompts and responses in the second dataset subset. These model-generated rankings are then combined with new, noisy labels (again, perturbed using Randomized Response) from the human data. A clever statistical technique called Maximum Likelihood Estimation (MLE) is used to combine these two sources of information, creating a more accurate, yet still private, set of labels. This refined set of labels is then used to further align M1, resulting in a more capable and privately aligned model, M2.
This progressive refinement is key. As the model evolves through stages, it becomes better at predicting preferences, reducing its reliance on the initial, heavily noisy human labels. This allows PROPS to achieve a better balance between privacy preservation and model performance.
Significant Performance Gains
The researchers conducted extensive experiments using various LLMs (Pythia-1B, GPT2-Large, GPT2-Medium) and datasets (AlpacaEval, Anthropic HH-RLHF, truthy-dpo-v0.1). The results are compelling: PROPS consistently outperforms both Randomized Response and DP-SGD, especially in ‘high-privacy regimes’ (where privacy requirements are very strict). For the same privacy budget, PROPS achieved up to 2.5 times higher win-rates compared to RR and up to 3 times higher win-rates compared to DP-SGD.
This means that with PROPS, LLMs can be aligned to human values more effectively while still rigorously protecting the privacy of the human labelers. The paper also provides theoretical guarantees, showing that PROPS is provably better than vanilla RR as long as the intermediate model gets better at predicting preferences.
While the paper primarily focuses on Direct Preference Optimization (DPO), the authors note that the PROPS framework can also be adapted for Reinforcement Learning with Human Feedback (RLHF), another common alignment method. This flexibility highlights the broad applicability of their approach.
Also Read:
- New Loss Function Enhances Language Model Alignment Stability
- Multi-Agent AI Teams Boost Privacy in Large Language Models
Looking Ahead
The development of PROPS marks a significant step forward in building more ethical and trustworthy LLMs. By providing a robust framework for preference-level privacy, it helps mitigate concerns about sensitive human feedback data. This work contributes positively to the field by promoting ethical considerations in LLM training and alignment, paving the way for broader adoption of privacy-preserving methods in language model development. You can find more details about this innovative work in the full research paper: PROPS: Progressively Private Self-alignment of Large Language Models.
