TLDR: Neo is a multi-agent testing framework designed to evaluate LLM-based agents by simulating realistic, human-like conversations. It uses a probabilistic state model to control dialogue flow, emotional tone, and topical intent, enabling dynamic and diverse test cases. Experiments show Neo can uncover edge cases and provide scalable, automated evaluation, significantly outperforming manual testing in speed and coverage, while approximating human testing effectiveness.
The rapid growth of Large Language Model (LLM)-based agents, now common in areas like customer service and financial advice, has highlighted a significant challenge: how do we effectively test these complex AI systems? Traditional testing methods, such as manual evaluations or static datasets, often fall short. Manual testing is slow, hard to repeat, and subjective, while static datasets quickly become outdated as LLM agents evolve. These limitations make it difficult to find subtle errors, security vulnerabilities, or conversational flaws that can undermine an AI system’s reliability and user trust.
Introducing Neo: A Smart Testing Framework
To address these issues, researchers have introduced Neo, a modular, multi-agent testing framework designed to make evaluating LLM-driven systems more robust, efficient, and adaptable. Neo aims to simulate realistic, human-like conversations to test LLM-based systems with greater coverage and flexibility. It integrates a probabilistic state model to control various aspects of a dialogue, including its flow, emotional tone, and the topic being discussed. This allows Neo to create dynamic and varied test cases across multiple turns of conversation.
Neo operates as a closed-loop, agent-driven simulation that mimics real user interactions and adaptively assesses agent behavior. It can be used for testing before an AI system is deployed and for monitoring it afterward. At its core, Neo consists of three main components: a Question Agent, an Evaluation Agent, and a Context Hub. These components interact with the Target Agent, which is the LLM-powered system being tested (e.g., a chatbot or a decision support system).
The Question Agent is responsible for generating realistic test inputs, like natural language prompts or queries, based on defined test objectives. It creates these inputs by pulling both static information (like domain details) and dynamic context (like conversation history or emotional tone) from the Context Hub. Once the Target Agent responds, the entire interaction is sent to the Evaluation Agent. This agent acts as an autonomous assessor, determining if the Target Agent’s response meets predefined criteria, such as being on-topic or appropriate. Currently, the Evaluation Agent provides a simple pass/fail signal, which then feeds back into the Context Hub, influencing how future questions are generated. For example, a failure might lead to the same question being rephrased, while a success might prompt a coherent follow-up.
The Context Hub serves as Neo’s central memory and control center. It stores all the necessary information that guides test behavior and coordination between agents, including configurations, prompt templates, and the history of interactions. This hub also maintains an evolving interaction state, which influences the generation of future test inputs and evaluation strategies. This design allows Neo to simulate user behavior over multiple turns in a way that is both diverse and responsive to the Target Agent’s performance.
Simulating Human Behavior with State Modeling
Neo’s testing approach is goal-driven and state-driven. Each turn in a test is guided by a structured “state vector” that captures interaction attributes like flow behavior, intent category, tone, and feedback from previous turns. This enables dynamic, multi-turn interaction patterns tailored to specific evaluation goals, such as security probing or realism testing. Instead of relying on static test cases, Neo generates adaptive test paths through probabilistic transitions over this state vector.
For conversational agents, the state is modeled as a combination of Flow Type (F), Intent Type (I), Tone Index (T), and Feedback (FB). The flow type dictates the structural direction of the conversation (e.g., starting a new topic, following up, or retrying). The intent governs the semantic target (e.g., routine, edge case, adversarial). The tone index simulates emotional progression, changing turn by turn. The feedback signal, derived from the Evaluation Agent’s judgment, modulates future state transitions, allowing Neo to mirror natural user behavior.
This state-driven framework allows Neo to generate a vast number of possible test sessions. The total number of unique states Neo can create depends on the number of discrete values each state dimension can take. For instance, with 3 interaction turns, 4 intent types, and 3 tone levels, Neo can theoretically produce over 10,000 unique test session variations, demonstrating its capacity for diverse and behaviorally rich testing.
Real-World Effectiveness: Experiments with a Chatbot
To evaluate Neo’s effectiveness, empirical studies were conducted using a production-grade Seller Financial Assistant chatbot. The experiments focused on two key objectives: Security Testing (malicious prompt injection and adversarial robustness) and Realism Testing (human-like multi-turn interactions).
In Experiment 1 (Security Testing), Neo was tasked with simulating adversarial behavior. It generated malicious questions designed to exploit or provoke undesired responses from the chatbot. Compared to human testers, Neo produced a balanced distribution across five attack types (Data Exfiltration, Privilege Escalation, Prompt Injection, TOS Violation/Threats, and Mixed Attack). Both human and Neo testers triggered failures primarily in the “Mixed Attack” category, indicating its complexity. While human testers induced a 5.8% break rate, Neo achieved a 3.3% break rate, showing that it can approximate human attack effectiveness and replicate complex vulnerabilities under controlled conditions.
Experiment 2 (Realism Testing) evaluated Neo’s ability to simulate realistic, multi-turn conversations. Neo generated sessions that showed highly human-like behavior, particularly in tone progression and emotional buildup, starting calm and escalating frustration when generic responses were received. Neo achieved 100% topic accuracy, consistently generating coherent, domain-specific questions. However, its emotional tone control was less consistent, with only about 55% of mid-range tones matching the intended sentiment. Additionally, about 32% of multi-turn sessions had unnatural or abrupt follow-up transitions, suggesting areas for improvement in tone modeling and conversational continuity.
Productivity and Scalability
One of Neo’s most significant advantages is its productivity and scalability. In the security testing experiment, six human testers spent approximately 5 hours generating 120 malicious test cases. Neo completed the same task in just 30 minutes (single-threaded), achieving a 10x–12x speedup. In the realism testing experiment, the time disparity was even greater: human testers required 20–30 minutes per session to craft coherent multi-turn conversations, while Neo generated 180 questions in roughly the same timeframe as the security test, maintaining high throughput regardless of task complexity.
Also Read:
- Automating Challenging Traffic Scenarios with LLM Agents
- HAMLET: Crafting Dynamic AI-Driven Theatre Experiences
Conclusion and Future Outlook
Neo addresses a critical need in modern LLM development: scalable, configurable, and intelligent testing of complex AI agent behaviors. It offers a systematic alternative to traditional QA, enabling the generation of diverse, multi-turn test interactions for proactive validation of functionality, safety, and user alignment at scale. While human testers remain essential for nuanced judgment, Neo provides consistency, breadth, and simulation control, making it a powerful complement.
Future directions for Neo include extending the Evaluation Agent beyond simple pass/fail to include multi-dimensional assessments (e.g., factual grounding, tone alignment), incorporating anonymized real-world interaction logs to fine-tune its generation strategies, and improving scalability through multi-threaded orchestration and lightweight LLM backends. Neo represents foundational infrastructure for rigorous, repeatable, and human-aligned testing of LLM agents, ultimately enhancing engineering productivity and the trustworthiness of AI systems. You can read the full research paper here.
