TLDR: A new diagnostic framework, ADM-ES, is introduced to evaluate and improve multi-agent LLM systems. It uses expert-annotated “golden” data, generates expanded “silver” datasets through AI-driven mutation, and employs an “Agent Judge” to score performance and suggest improvements. Applied to a recruiter-assistant AI, the framework successfully identified cognitive failures like biased phrasing and extraction errors, while also guiding agents to adopt expert-level reasoning and style, moving beyond traditional static evaluations.
The rapid advancement of artificial intelligence, particularly in large language models (LLMs), has led to the creation of sophisticated AI agents capable of complex tasks like planning, using external tools, and even self-reflection. However, these advanced systems, often operating with multiple interconnected agents, present a unique challenge: how do we accurately evaluate and improve them when their behavior can be unpredictable and their decision-making processes involve many steps?
Traditional evaluation methods, designed for simpler, more deterministic AI models, fall short. They can’t effectively diagnose issues like biased language, errors in extracting information, or incorrect tool usage that emerge in these dynamic, multi-agent environments. This gap in evaluation is precisely what a new diagnostic framework, the Agent Diagnostic Method for Expert Systems (ADM–ES), aims to address.
A New Approach to AI Evaluation
Developed by Andrejs Sorstkins, Dr Alistair Baron, and Josh Bailey, the ADM–ES framework moves beyond simply scoring performance. It’s designed to actively diagnose and facilitate the transfer of expert human behavior into LLM-powered agents. The framework integrates three key components:
- Curated ‘golden’ datasets: These are high-quality examples of expert annotations, where human specialists have meticulously identified and corrected issues.
- ‘Silver’ datasets: These are generated through a process called ‘controlled behavioral mutation,’ where an AI model learns from the golden examples to create a much larger dataset of expert-like responses.
- An LLM-based ‘Agent Judge’: This AI acts as a critic, scoring agent performance and prescribing specific improvements.
The ADM–ES pipeline involves four main stages: First, experts create the ‘golden’ dataset. Second, an ‘Agent Mutator’ uses these golden examples to generate a broader ‘silver’ dataset, effectively scaling expert knowledge. Third, the ‘Agent Judge’ evaluates the system’s outputs using two specialized diagnostics: the Text Block Extraction Diagnostic (ED) and the Behaviour Diagnostic (BD). Finally, the Judge’s recommendations are organized into a ‘Recommendation Map’ to identify recurring improvement themes.
Diagnosing Agent Performance: ED and BD
The **Text Block Extraction Diagnostic (ED)** focuses on how accurately the AI system extracts relevant information from a given text, comparing its selections to those made by human experts. It measures precision, recall, and F1-score at the sentence level, ensuring the AI identifies the correct critical pieces of information.
The **Behaviour Diagnostic (BD)**, on the other hand, evaluates the AI’s overall style, tone, reasoning clarity, and adherence to expert-level communication. It compares the AI’s generated responses to the expert-like ‘silver’ data, assessing aspects like factual sufficiency, reasoning clarity, tone/style match, and tool-use fidelity.
Real-World Application: JobFair’s Recruiter-Assistant System
The framework was put to the test on JobFair Software’s multi-agent recruiter-assistant system, which uses AI to optimize job descriptions for inclusive language. The study specifically focused on two specialist agents: the Gendered Language Agent (GLA) and the Neurodiversity Agent (NDA).
The results from the ‘Agent Mutator’ stage were encouraging. For both agents, the mutation process significantly improved the alignment of their ‘Comment Suggestion’ outputs (which are more detailed, rewrite-like options) with expert style. The Neurodiversity Agent, in particular, showed substantial gains in adopting expert-like phrasing and intervention strategies. This demonstrates that the framework can effectively transfer desired behaviors from human experts to AI agents.
However, the ‘Extraction Diagnostic’ for the Neurodiversity Agent revealed areas for improvement. While the agent was good at using correct terminology and maintaining detail accuracy when it did extract information, it often missed a significant portion of expert-identified evidence. This suggests the agent was ‘careful but timid,’ needing to improve its completeness and recall of critical information.
The ‘Behaviour Diagnostic’ showed that the Neurodiversity Agent maintained strong semantic alignment and tone match with expert standards. There was a slight ‘underfit’ in stylistic fidelity and manner of expression, meaning the agent sometimes simplified sentence structures or rhetorical devices compared to experts. Crucially, the Agent Judge’s prescriptions were actionable, providing concrete suggestions for refining prompts and retrieval mechanisms to guide the system towards the desired behavioral profile.
Also Read:
- AI Learns to Fix C++ Code: A New Approach to Compilation Repair
- AI-Powered Root Cause Analysis for Complex Microservices
The Recommendation Map: A Blueprint for Improvement
One of the most innovative aspects of the ADM–ES framework is the ‘Recommendation Map.’ By embedding and clustering the Agent Judge’s prescriptions, the framework creates a visual and structural aggregation of recurring failure patterns and effective intervention strategies. For instance, the ED prescriptions clustered into themes like ‘tighter evidence inclusion criteria’ and ‘salience re-weighting,’ while BD prescriptions formed clusters around ‘tone normalization’ and ‘structured, stepwise reasoning scaffolds.’ This map serves as a reusable knowledge structure, allowing teams to systematically audit system weaknesses and track improvements over time.
This research offers a viable blueprint for diagnosing and systematically improving stochastic, tool-augmented LLM agents. By transforming evaluation into an active, iterative process of expert knowledge propagation, the ADM–ES framework helps ensure that AI systems not only perform tasks but also do so with expert-level reasoning and style. You can read the full paper here: Diagnostics of cognitive failures in multi-agent expert systems using dynamic evaluation protocols and subsequent mutation of the processing context.
