TLDR: This research introduces “compositional privacy leakage,” a new risk in multi-agent AI systems where individually harmless information, when combined from different agents, reveals sensitive data. The paper proposes two defenses: Theory-of-Mind (ToM), where agents anticipate adversarial intent, and Collaborative Consensus Defense (CoDef), where agents collectively vote to block risky queries. Experiments show CoDef offers the best balance, significantly reducing leakage while maintaining useful functionality, highlighting the need for coordinated privacy safeguards in collaborative AI.
As large language models (LLMs) become increasingly integrated into multi-agent systems, new and complex privacy risks are emerging. These threats go beyond simple memorization of data or direct inference from a single interaction. A recent research paper, “The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaboration”, by Vaidehi Patil, Elias Stengel-Eskin, and Mohit Bansal, delves into a phenomenon they term ‘compositional privacy leakage’. This occurs when seemingly harmless responses, when combined across multiple interactions, can cumulatively allow adversaries to uncover sensitive information.
The core idea is that no single piece of information shared by an individual AI agent might be sensitive on its own. However, an adversary, by querying different agents and piecing together these fragments, can infer private attributes that were never explicitly disclosed by any one agent. Imagine an adversary obtaining a customer ID-to-name mapping from one agent, product purchase logs from another, and insurance claim information from a third. While each piece is innocuous in isolation, their combination could reveal sensitive health information, such as a customer self-monitoring for heart issues after purchasing specific medical devices, even if they have no diagnosed condition.
This type of privacy risk is distinct from traditional memorization, where models might regurgitate sensitive training data, or single-model risks. Instead, compositional leakage arises dynamically during interaction, through cross-agent context accumulation and collaborative inference, often when an adversary has some background knowledge.
A Framework for Understanding Leakage
To systematically study this threat, the researchers developed a framework that models how auxiliary knowledge held by an adversary and the interactions between agents jointly amplify privacy risks. They created controlled scenarios where sensitive attributes are explicitly split across different entities, ensuring that no single agent possesses enough information to infer the sensitive data alone. Success in these scenarios necessarily comes from an adversary composing responses from multiple sources.
Introducing Defense Strategies
The paper proposes and evaluates two innovative defense strategies to counter compositional privacy leakage:
- Theory-of-Mind (ToM) Defense: In this approach, defender agents are designed to infer the questioner’s intent. They anticipate how their outputs might be exploited by adversaries and, based on this reasoning, decide whether to withhold information. This involves the defender simulating the adversary’s knowledge evolution and blocking queries that could enable sensitive inferences.
- Collaborative Consensus Defense (CoDef): This strategy involves responder agents collaborating with their peers. They share aggregated contextual information and individually vote on whether a query is safe to answer. The final decision to allow or block a query is made under a consensus rule, where even a single defender’s decision to block is sufficient to deny the query. This enables collective mitigation of compositional privacy leakage while aiming to preserve benign functionality.
Also Read:
Key Findings and Trade-offs
The experiments, conducted using various LLMs like Qwen3-32B, Gemini-2.5-pro, and GPT-5 as defenders, revealed significant insights:
- Limited Protection from Chain-of-Thought (CoT): Simple step-by-step reasoning (Chain-of-Thought) alone offered limited protection, with only about 39% of sensitive queries being blocked on average. While CoT maintained good utility for benign tasks, it left systems highly vulnerable to compositional leakage.
- ToM’s Strength and Weakness: The Theory-of-Mind defense substantially improved sensitive query blocking, raising rates to as high as 97%. However, this often came at a cost to benign utility, meaning it sometimes over-blocked legitimate requests, reducing the success rate of harmless tasks.
- CoDef’s Balanced Approach: The Collaborative Consensus Defense achieved the best balance between privacy and utility. It blocked a high percentage of sensitive queries (86-90%) while maintaining higher success rates for benign tasks. This highlights the benefit of combining explicit reasoning with collaboration among defenders.
- Model Capabilities Matter: More capable models like GPT-5 showed stronger balanced outcomes, suggesting an inherent ability to better recognize adversarial goals and sustain benign reasoning chains.
The research underscores that privacy risks in multi-agent systems cannot be fully addressed by isolated safeguards. Robust protection requires coordination, shared reasoning, and explicit modeling of adversarial strategies. This work provides a crucial step towards systematically evaluating these dynamics and offers actionable insights for designing safeguards against compositional, context-driven privacy leakage in the increasingly collaborative world of AI.
