Unmasking AI’s Data Habits: A Deep Dive into Chatbot Privacy Policies

TLDR: A new research paper analyzes the privacy policies of six major U.S. AI developers (Amazon, Anthropic, Google, OpenAI, Meta, Microsoft) to understand how they use user chat data for model training. It reveals that all six train on user inputs by default, some retain data indefinitely, and personal/sensitive information, including children’s data, is often collected. The study highlights a lack of transparency, the power of default settings, and the blurring of product boundaries, concluding with recommendations for federal privacy regulation, opt-in consent for training, improved transparency, default data filtering, and promotion of privacy-preserving AI innovations.

Large language models (LLMs) have become an integral part of daily life, with hundreds of millions of people interacting with chatbots regularly. As developers race to enhance model capabilities and capture market share, the demand for high-quality training data is soaring. A recent research paper delves into the privacy policies of six leading U.S. AI developers to understand how they utilize user conversations for model training.

Understanding the Privacy Landscape

For decades, privacy policies have been the primary, albeit flawed, method for companies to inform consumers about data collection and usage. These documents are often dense, complex, and rarely read or fully understood by users. With the rapid proliferation of LLM-powered chatbots, these policies remain the main resource for consumers seeking clarity on how their interactions are processed.

The paper, titled “User Privacy and Large Language Models: An Analysis of Frontier Developers’ Privacy Policies” by Jennifer King, Kevin Klyman, Emily Capstick, Tiffany Saade, and Victoria Hsieh from Stanford University, highlights a critical gap in understanding. While concerns about data scraped from the internet for LLM training have been voiced, less attention has been paid to data collected directly from consumer interactions with LLMs. This is particularly relevant given the explosive growth in chatbot usage, exemplified by ChatGPT’s rapid adoption.

Key Findings on Data Practices

The researchers analyzed the privacy policies of Amazon, Anthropic, Google, OpenAI, Meta, and Microsoft, focusing on how they disclose the collection and use of chat data. Their findings reveal several concerning trends:

• Default Data Use: All six developers appear to use users’ chat data to train and improve their models by default. This means that unless users actively opt-out, their conversations contribute to model development.
• Indefinite Retention: Some companies retain this chat data indefinitely, raising significant data security and privacy risks.
• Sensitive Information: Developers may collect and train on personal information disclosed in chats, including sensitive data like biometric and health information, as well as files uploaded by users.
• Children’s Data: Four of the six companies examined appear to include children’s chat data for model training. While Anthropic does not accept users under 18, and Microsoft does not train Copilot on data from authenticated users under 18, Google expanded Gemini to include children under 13 and trains on data from 13-18 year olds if they opt-in. Amazon, Meta, and OpenAI do not explicitly differentiate, suggesting they may train on data from users aged 13 and older by default.
• Lack of Transparency: Developers’ privacy policies often lack essential information, necessitating users to navigate a web of documents (main policies, sub-policies, FAQs) to understand practices fully. OpenAI, for instance, relied on at least six separate policies.
• Other Data Sources: Beyond chat inputs, developers also draw on uploaded files, voice data, images, and data from other products on their platforms (e.g., Microsoft using data from Bing, Meta using data from other Meta products) for training.
• Human Review: Google and OpenAI explicitly mention using human reviewers for user chats, advising users not to enter confidential information.
• De-identification Efforts: Only Microsoft explicitly states efforts to remove identifiable information like names and sensitive personal data before training. Anthropic and OpenAI mention rules-based approaches to prevent disclosure of incidentally captured personal data.
• Personalization: Google, Meta, Microsoft, and OpenAI offer personalized chatbot experiences that persist across sessions, recalling user details and preferences.

Implications and Challenges

The study highlights the tension between data protection principles and the insatiable demand for data to train LLMs. The blurring boundaries between chatbots and other platform products challenge principles like data minimization (collecting only necessary data) and purpose limitation (using data only for its stated purpose).

The power of defaults is a significant factor; most users do not change default settings, meaning the default collection of chat data for model training has far-reaching privacy implications. The conversational nature of chatbots encourages deeper personal disclosures, making the risks of data leakage and indefinite retention even more pronounced. The paper notes that LLMs are known to memorize and regurgitate personal data, posing risks even to non-users whose data might be in publicly scraped training sets.

Furthermore, the justification for these practices often relies on claims of societal benefit and improving the model for everyone, which can be seen as a form of ‘guiltshaming’ to encourage data sharing without fully disclosing the potential risks to users.

Also Read:

• Beyond Outputs: Unmasking LLM Training Data Through Internal ‘Neural Breadcrumbs’
• Unpacking Human Agency: A New Benchmark Reveals AI’s Impact on Our Control

Recommendations for a More Private AI Future

To address these challenges, the researchers propose several recommendations:

1. Comprehensive Federal Privacy Regulation: The U.S. needs baseline consumer privacy regulations that set limits on personal data collection and use, especially for LLMs, to overcome the current fragmented state of state-level laws.
2. Require Opt-in for Model Training: Policymakers should mandate that developers obtain explicit opt-in consent from users for model training, rather than making it the default. Children should never be opted-in by default.
3. Improve Transparency: Developers must clarify how they collect, store, and process data for model training using standardized, user-friendly methods. This includes revamping privacy policies and releasing AI-specific documentation like datasheets and model cards.
4. Filter Personal Information by Default: Developers should proactively filter sensitive personal data from chat inputs and uploaded files by default. User interfaces should also make it easy for users to edit, delete, or strip information from their chats.
5. Promote Privacy-Preserving AI Innovation: Companies should adopt and build upon existing privacy-enhancing technologies and approaches, such as temporary chats, on-device models, no-logs policies, and secure hardware enclaves, instead of treating user privacy as an afterthought.

The paper concludes by emphasizing that while LLMs offer transformative potential, this must not come at the cost of user privacy. A transparent and trusting relationship with the public is essential for the widespread acceptance and ethical development of AI. You can read the full research paper here.