TLDR: A research paper systematically investigates privacy risks in LLM Knowledge Distillation (KD), finding that student models inherit significant membership and memorization privacy risks from their teachers, even when trained on public data. The extent of risk varies by KD technique, student size (smaller students have less risk but lower utility), loss function (Reverse KL helps), and the use of student-generated data (increases risk). The study also reveals that memorization and membership risks differ across NLP tasks and that privacy leakage varies significantly across individual transformer blocks within an LLM.
Large Language Models (LLMs) have become incredibly powerful, but their immense size often comes with high computational costs. To address this, a technique called Knowledge Distillation (KD) is widely used. KD involves transferring knowledge from a large “teacher” LLM to a smaller, more efficient “student” model. This process is often assumed to also offer privacy benefits, as the student model is typically trained on public data, not the private data the teacher was exposed to.
However, a recent research paper titled “Membership and Memorization in LLM Knowledge Distillation” challenges this assumption. The study, conducted by Ziqi Zhang, Ali Shahin Shamsabadi, Hanxiao Lu, Yifeng Cai, and Hamed Haddadi from institutions including Peking University, Brave Software, Purdue University, and Imperial College London, systematically investigates the privacy risks that student models might inherit from their teachers. You can find the full paper and more details on their findings by visiting the research paper link.
The researchers explored six different LLM Knowledge Distillation techniques across seven natural language processing (NLP) tasks, using three teacher model families (GPT-2, LLAMA-2, and OPT) and various student model sizes. Their comprehensive analysis revealed a critical insight: all existing LLM KD approaches carry significant membership and memorization privacy risks from the teacher to its students. This means that even if a student model is trained on public data, it can still inadvertently reveal information about the private data its teacher was trained on.
Understanding the Privacy Risks
The paper focuses on two main types of privacy leakage: membership privacy and memorization privacy.
Membership Privacy: This refers to the ability to infer whether a specific data record was part of the teacher model’s private training dataset. The study used seven state-of-the-art Membership Inference Attacks (MIAs) to test this. They found that MIAs could successfully infer membership information from student models, with some attacks achieving high accuracy (e.g., AUC scores over 0.80 for OPT models).
Memorization Privacy: This concerns the student model’s ability to reproduce verbatim (word-for-word) parts of the teacher’s private training data. The study found that student models could memorize a non-trivial percentage of the same samples that the teacher memorized, with 11.35% of samples being memorized to the same extent by both teacher and student.
Key Findings on KD Techniques and Factors
While all KD techniques showed privacy risks, the extent of these risks varied. Techniques like SeqKD and ImitKD exhibited higher privacy leakage. This is attributed to SeqKD using the teacher’s exact output to build its training data, which might inadvertently include private information the teacher had memorized.
The size of the student model also plays a role. Smaller student models generally resulted in lower attack performance, meaning less privacy leakage. However, this comes with a trade-off: smaller models also tend to have lower utility, meaning they perform less effectively on downstream tasks. This highlights a crucial privacy-utility-efficiency trade-off in LLM KD.
The choice of loss function during KD also impacts privacy. Using a Reverse KL (RKL) loss function was found to mitigate membership inference attacks compared to the standard Kullback-Leibler (KL) divergence. Conversely, increasing the ratio of student-generated data in the training process was shown to increase privacy risk.
Memorization vs. Membership: A Discrepancy
An interesting discovery was the significant disagreement between memorization and membership privacy risks across different NLP tasks. For instance, tasks like creative writing, general question answering, and brainstorming showed the highest membership privacy risk. In contrast, classification and closed question answering tasks exhibited the highest memorization. This suggests that these two types of privacy leakage are distinct and require different considerations.
Per-Block Privacy Analysis
Taking a novel approach, the researchers also analyzed privacy leakage at a more granular level: per transformer block within the LLM. They found that privacy leakage varies significantly across different blocks of the same LLM. For example, in GPT2-Large, certain deep blocks were more vulnerable, while in OPT-1.3B, middle blocks showed higher vulnerability. This fine-grained analysis could pave the way for designing more privacy-friendly KD strategies by selectively mitigating risks in specific components of the teacher model.
Also Read:
- Optimizing Language Model Compression with Selective Reflection Distillation
- New Strategies for Preventing Unintended AI Behavior During Training
Implications for LLM Development
The findings of this research underscore that Knowledge Distillation, while beneficial for efficiency, does not inherently guarantee privacy protection for LLMs. Developers and researchers need to be aware of these inherited privacy risks and consider them when deploying distilled models, especially when the teacher model has been trained on sensitive or private data. The study provides a crucial empirical foundation for understanding these risks and highlights areas for future research in developing more robust privacy-preserving KD techniques.
