TLDR: MAFA (Multi-Agent Framework for Annotation) is a production-deployed AI system by JP Morgan Chase that automates and enhances enterprise-scale data annotation. It uses specialized AI agents, structured reasoning, and a judge-based consensus mechanism to process millions of customer utterances, eliminating backlogs and saving over 5,000 hours of manual work annually. The configurable framework dynamically adapts to various annotation tasks, achieving high agreement with human annotators and significantly outperforming traditional and single-agent baselines, while seamlessly integrating human oversight for complex cases.
Large enterprises today face a significant hurdle in managing the vast amounts of customer interactions that flow through their digital channels. For companies like JP Morgan Chase, millions of customer utterances each month require accurate categorization for various applications, from intent classification to FAQ mapping and entity extraction. Traditionally, this has relied on human teams, a method that has proven unsustainable, leading to growing backlogs and delays in improving AI models and customer experience.
Addressing this critical challenge, JP Morgan Chase has developed and deployed MAFA (Multi-Agent Framework for Annotation), a sophisticated system designed to transform enterprise-scale annotation workflows. MAFA leverages configurable multi-agent collaboration to achieve human-level annotation quality at an unprecedented scale.
The Annotation Challenge in Financial Services
Before MAFA, JP Morgan Chase’s annotation process involved five full-time human annotators handling approximately 3,000 utterances daily. This was a severe mismatch for a daily volume often exceeding 30,000, resulting in a backlog of over a million unannotated utterances. This manual approach was not only costly but also suffered from quality inconsistencies, with inter-annotator agreement averaging only 72%. Such delays hindered innovation and impacted customer experience.
How MAFA Works: A Collaborative AI Approach
MAFA is built on a hierarchical multi-agent architecture. It combines specialized AI agents with structured reasoning and a judge-based consensus mechanism. Unlike single-model approaches, which often lack the nuance and reliability needed for production, MAFA’s multi-agent system ensures greater consistency and accuracy.
The framework is highly configurable, allowing organizations to define custom annotation types (such as FAQs, intents, entities, or domain-specific categories) through simple configuration files rather than complex code changes. This dynamic task adaptation is a key innovation.
The system’s architecture includes:
- Query Planning Agent: This entry point analyzes and expands ambiguous customer queries, using advanced models like OpenAI’s GPT-4o. For instance, a query like “cash back” might be expanded to include “cash back policies, rewards, redemption, credit cards” to ensure comprehensive matching.
- Specialized Annotation Agents: MAFA deploys four complementary agents, each with distinct retrieval strategies. Some agents rely on structured prompting for direct matching based on primary or full context fields, while others leverage dense vector representations and embedding models for semantic similarity.
- Judge Agent: This acts as the final arbiter, synthesizing recommendations from all other agents into an optimized ranking. It uses a weighted voting scheme, prioritizing high-confidence predictions and considering domain-specific business rules.
A crucial aspect of MAFA is its structured agent prompting, which guides agents through systematic annotation decisions using JSON-based prompts. This ensures consistent output and provides traceable reasoning steps, vital for auditing and debugging in a regulated environment like financial services.
Tangible Business Impact and Efficiency Gains
Since its deployment at JP Morgan Chase in May 2025, MAFA has processed over 1.1 million utterances and eliminated a 1 million-utterance backlog. It has achieved, on average, 86% agreement with human annotators, annually saving over 5,000 hours of manual annotation work. The system processes utterances with high confidence classifications (typically 85% high, 10% medium, 5% low), allowing human annotators to focus exclusively on ambiguous and low-coverage cases.
MAFA has demonstrated significant improvements over traditional and single-agent annotation baselines, showing 13.8% higher Top-1 accuracy, 15.1% improvement in Top-5 accuracy, and 16.9% better F1 score in internal intent classification datasets, with similar gains on public benchmarks.
The system seamlessly integrates with existing human workflows through a confidence-based routing strategy. High-confidence annotations are auto-accepted, medium-confidence ones are flagged for periodic audits, and low-confidence cases are routed for human review, maximizing efficiency while maintaining quality.
Also Read:
- MASC: Equipping Multi-Agent LLM Systems with Real-Time Self-Correction
- Helmsman: Automating Federated Learning System Design with Multi-Agent Collaboration
Scalability and Future Outlook
MAFA is deployed on a Kubernetes cluster, demonstrating linear scalability and cost efficiency, achieving an 85% reduction in cost per query compared to a single-agent call. It handles 8,000 daily queries with 99.7% uptime.
The success of MAFA at JP Morgan Chase provides a blueprint for other organizations facing similar annotation challenges. It highlights that multi-agent architectures offer inherent resilience, configurability is essential for enterprise adoption, and transparent confidence scoring fosters trust and collaboration between AI and human teams. This work bridges the gap between theoretical multi-agent systems and practical enterprise deployment, paving the way for robust, enterprise-ready AI solutions. You can read the full research paper here.
