TLDR: A new research paper introduces a differentially private in-context learning (DP-ICL) framework that integrates nearest neighbor search (kNN) with privacy filters. This method addresses the overlooked privacy risks in context data retrieval for large language models (LLMs). By intelligently selecting relevant examples while tracking privacy costs, it significantly outperforms existing DP-ICL baselines in text classification and document question answering, achieving better privacy-utility trade-offs.
In-context learning (ICL) has become a cornerstone for tailoring large language models (LLMs) to specific tasks without the need for extensive retraining. It works by providing the LLM with a few examples directly within the prompt, guiding its response to a new query. This approach is popular because it’s efficient and often only requires API access to the LLM.
However, the convenience of ICL comes with significant privacy concerns. LLMs are known to sometimes memorize training data, leading to potential leakage of sensitive information. While research has addressed privacy in LLM fine-tuning and pretraining, a critical aspect of modern LLM pipelines has often been overlooked: the similarity search used to retrieve relevant context data for ICL.
Existing methods for Differentially Private In-Context Learning (DP-ICL) typically fall into two categories: generating synthetic examples privately or using a “pay-per-use” model where privacy costs accumulate with each query. While these methods offer privacy guarantees, they often face challenges such as computational expense, reliance on specific LLM outputs (like logits), or the use of randomly sampled examples which can degrade performance.
The problem with random sampling is that the quality of ICL outputs is highly sensitive to the examples provided. Randomly chosen examples can lead to increased uncertainty and poorer performance, sometimes even worse than a “zero-shot” prediction (where no examples are given). This highlights the importance of effective example selection.
A new research paper, titled “Differentially Private In-Context Learning with Nearest Neighbor Search,” introduces a novel framework that addresses this gap. Authored by Antti Koskela, Tejas Kulkarni from Nokia Bell Labs, and Laith Zumot from Nokia, the paper proposes integrating nearest neighbor search (kNN) into DP-ICL in a privacy-aware manner. You can read the full paper here: Differentially Private In-Context Learning with Nearest Neighbor Search.
The core innovation lies in replacing random example selection with kNN retrieval. Instead of picking examples randomly, the method identifies the ‘k’ most similar examples from a database for each query. To maintain privacy, this retrieval is combined with a “privacy filter.” This filter meticulously tracks the cumulative privacy cost of selected samples, ensuring that a central differential privacy budget is strictly adhered to.
The researchers employ a rigorous privacy accounting method called individual Rényi Differential Privacy (RDP) filtering. This allows for adaptive selection of mechanisms while ensuring privacy guarantees. For the nearest neighbor search itself, they initially focus on a “FLAT” index, which performs an exhaustive search and simplifies privacy accounting due to its inherent stability. They also discuss future avenues for extending this to more approximate, but faster, indexing methods like IVF or HNSW using techniques like DP k-means.
Experimental results demonstrate a substantial advantage of this new kNN-based DP-ICL method. Across various benchmarks, including text classification (AGNews and TREC datasets) and document question answering (Federated DocVQA and SQuAD v1.1), the proposed approach significantly outperforms existing baselines. It achieves more favorable privacy-utility trade-offs, meaning it can provide strong privacy guarantees without sacrificing much in terms of performance.
For text classification, using models like OPT-1.3B, the kNN method showed higher test accuracies compared to baselines, especially at higher privacy budgets (epsilon values). In document question answering, evaluated with Llama3.3-70B-it and Gemini-1.5-flash-8B, the DP-KSA-kNN method consistently yielded better ANLS, exact match, ROUGE, and BLEU scores than the DP-KSA baseline and zero-shot predictions.
Also Read:
- Unlocking AI Transparency: New ‘Locality Dial’ Controls Interpretability in Language Models
- Collaborative LLM Inference: Introducing Federated Attention for Edge Networks
This work represents a crucial step forward in making LLM applications both powerful and privacy-preserving, particularly in scenarios where sensitive data is used for in-context learning. By intelligently selecting relevant examples while strictly adhering to privacy budgets, this method paves the way for more secure and effective AI systems.
