TLDR: A Salesforce research study, published in May 2025, found that AI customer service agents exhibit ‘near-zero confidentiality awareness’ when handling customer data, posing significant privacy risks. The study also highlighted the agents’ limited effectiveness, with success rates of only 58% for single-turn tasks and a mere 35% for multi-turn interactions, indicating a substantial gap between current AI capabilities and real-world customer service demands.
A recent study conducted by Salesforce researchers has unveiled a critical vulnerability in the design of AI customer service agents: a profound lack of data confidentiality awareness. The research, published in May 2025, indicates that these AI agents exhibit ‘near-zero confidentiality awareness’ when accessing and processing sensitive customer information from databases. This finding raises significant concerns about data privacy and security in AI-driven customer support systems.
Beyond the confidentiality issues, the Salesforce study also casts doubt on the current efficacy of AI agents in handling complex customer service interactions. The research revealed that even leading large language model (LLM) agents achieved only a 58% success rate in single-turn scenarios, which involve simple, direct questions. Performance plummeted further in multi-turn settings, where agents had to manage multiple inputs and maintain context, achieving an approximate success rate of just 35%. This suggests a substantial gap between current LLM capabilities and the intricate demands of real-world enterprise customer service.
The study highlighted that the most advanced models, such as OpenAI’s o1, were required to achieve even these modest multi-turn success rates, leading to potentially prohibitive costs—estimated at $3-4 for each simple customer service exchange. This economic factor, combined with the performance limitations and critical confidentiality flaws, challenges the widespread assumption that AI agents are ready to fully replace human customer service representatives.
Also Read:
- MIT Study Reveals High Failure Rate in Corporate Generative AI Pilot Programs
- The Rise of Custom AI Agents: Why They Outperform General-Purpose Models Like ChatGPT for On-Device Applications
The researchers’ conclusion points to a ‘significant gap between current LLM capabilities and the multifaceted demands of real-world enterprise scenarios.’ Another paper from Microsoft reportedly corroborates these findings, suggesting that LLMs struggle to recover once they miss a step or make a mistake in a multi-turn sequence. These findings underscore the urgent need for significant breakthroughs in AI architecture and ethical design to ensure both the effectiveness and security of AI customer service solutions before widespread adoption.
