TLDR: A new AI system for customer support agents generates real-time, concise summary notes during conversations, using a fine-tuned Mixtral-8x7B model and a DeBERTa classifier. It incorporates agent feedback for continuous improvement, leading to a 3% reduction in case handling time (up to 9% for complex cases) and high agent satisfaction, without negatively impacting customer satisfaction.
Customer support interactions can be complex and time-consuming, often involving lengthy conversations across multiple channels like phone, chat, and email. A significant portion of an agent’s time, estimated at around 10%, is spent on writing summary notes. These notes are crucial for handoffs, compliance, and quality audits, but the process of creating them can lead to increased context-switching and overall longer handling times. Two main challenges exacerbate this issue: the sheer volume of unstructured text from multi-source inputs, making it hard to pinpoint critical information, and the demand for high accuracy in summaries to ensure effective issue resolution and customer satisfaction.
To address these challenges, researchers have introduced an innovative real-time incremental summarization system designed to enhance efficiency and reduce agents’ context-switching efforts. This system focuses on generating concise bullet notes during conversations, rather than waiting until the end. The core of this approach lies in two key innovations: a Progressive Note-Taking with Quality Control workflow and an Agent-Edits Learning Framework.
The Progressive Note-Taking workflow intelligently determines the optimal moments to generate notes. It utilizes a fine-tuned Mixtral-8x7B large language model (LLM) for continuous note generation. This LLM is trained to produce only new, incremental bullets from recent dialogue turns, ensuring that previously generated and agent-accepted notes are preserved. Following the LLM, a DeBERTa-based classifier acts as a quality control mechanism, filtering out trivial or redundant content and retaining only high-value utterances, such as customer issues, agent solutions, or follow-up questions. This ensures the summaries remain concise and relevant.
The Agent-Edits Learning Framework is a crucial component that integrates human feedback into the system’s continuous improvement. When agents review the generated summary bullets in their user interface, they can correct errors or add missing details. These real-time edits immediately update the online notes, ensuring subsequent LLM prompts use the most accurate, agent-verified information. Offline, these agent edits are logged and used to refine the model further. A quality assessment process, involving both LLM-based evaluation and human review, identifies high-quality edits. These validated ‘before-edit’ and ‘after-edit’ pairs are then used to fine-tune the summarization LLM through supervised fine-tuning and preference alignment techniques, closing the feedback loop and continuously improving summary quality.
The system’s effectiveness was rigorously evaluated through both offline and online experiments. Offline metrics assessed summary quality based on conciseness, completeness, and truthfulness, using an LLM-judge (GPT-4o) which proved to be superior to human annotators in performance and significantly faster. The results showed clear incremental gains from the base Mixtral model to the fine-tuned version without feedback, and further improvements with the integration of agent edit feedback. An ablation study also confirmed that the bullet classifier significantly improved conciseness and overall summary quality by reducing length by approximately 26.5% on average.
In real-world deployment, the system demonstrated substantial business impact. Online experiments, conducted using a Diff-in-Diff quasi-experiment across over 92% of production cases, revealed an overall 3% reduction in agent working time. For highly complex cases, involving three or more agents and typically longer handling times, the reduction was even more significant, reaching up to 9%. These time savings are attributed to reduced manual note-writing and easier access to comprehensive, model-generated notes. Agent satisfaction surveys across English, French, and Spanish-speaking cohorts reported high satisfaction rates, with 95.2% in English, 81.8% in French, and 89.4% in Spanish agents reporting being ‘Satisfied’ or ‘Very Satisfied’. Crucially, customer Net Promoter Scores (NPS) remained neutral, indicating no negative impact on customer satisfaction.
Learnings from the deployment also highlighted optimizations in model serving, such as using TensorRT-LLM for reduced inference latency. Multilingual performance presented specific challenges, with French summaries showing lower truthfulness due to complex number structures and Spanish summaries having lower completeness in capturing agent solutions due to varied verb conjugations. Despite these, the system has proven to be a valuable tool, saving an estimated hundreds of thousands to millions of agent hours annually. Future work aims to expand language support, incorporate multimodal contexts, and continuously retrain the classifier based on ongoing agent feedback.
Also Read:
- Empowering Customer Support: How Airbnb’s Agent-in-the-Loop System Drives Continuous AI Improvement
- Enhancing LLM Performance for Business Conversation Summarization with Domain-Adaptive Pre-Training
This incremental summarization system represents a significant advancement in leveraging AI to improve operational efficiency and agent experience in customer support, demonstrating how continuous feedback and intelligent note-taking can effectively enhance summary quality and agent productivity at scale. You can read the full research paper here: Incremental Summarization for Customer Support via Progressive Note-Taking and Agent Feedback.
