TLDR: AgentArch is a benchmark evaluating 18 AI agent architectures across state-of-the-art LLMs on two enterprise tasks. It reveals that optimal architectures are model and task-specific, challenging a “one-size-fits-all” approach. Function calling generally outperforms ReAct, and thinking tools help non-reasoning models on simpler tasks. Overall agentic performance on complex enterprise tasks remains low (max 35.3% success), highlighting significant weaknesses and reliability issues in current LLMs for real-world business applications.
In the rapidly evolving landscape of artificial intelligence, large language models (LLMs) are moving beyond simple text generation to become autonomous agents capable of complex decision-making and task completion. This shift holds immense promise for enterprise automation, where AI agents can integrate with existing systems to achieve multi-step objectives with minimal human intervention.
However, despite this growing interest, practitioners face significant challenges. Much of the existing research evaluates individual components of agentic architectures in isolation, such as orchestration strategies or prompting approaches, without considering how these elements interact within complex multi-agent systems. Furthermore, while many benchmarks exist, few focus specifically on the unique constraints and requirements of enterprise workflows, which demand high reliability and seamless integration with business systems.
To address these critical gaps, ServiceNow researchers Tara Bogavelli, Roshnee Sharma, and Hari Subramani introduced AgentArch, a comprehensive enterprise-specific benchmark. This study systematically evaluates 18 distinct agentic configurations across state-of-the-art large language models, providing much-needed empirical guidance for developers building AI systems for business applications. You can read the full paper here: AgentArch: A Comprehensive Benchmark to Evaluate Agent Architectures in Enterprise.
Key Architectural Dimensions Explored
The AgentArch benchmark delves into four crucial dimensions of agentic systems:
- Orchestration Strategy: This examines how agents coordinate, comparing single-agent systems to multi-agent approaches like orchestrator-led isolated agents (where a central orchestrator manages all communication) and orchestrator-led open agent networks (where agents communicate directly).
- Agent Prompt Implementation: The study compares two prevalent agent styles: function calling (where the model directly selects tools) and ReAct (Reasoning and Acting), which involves explicit intermediate reasoning before selecting actions.
- Memory Architecture: This dimension explores how information is shared among agents, contrasting complete memory (where all previous tool calls and responses are visible) with summarized memory (where agents receive only final summaries).
- Thinking Tool Integration: The benchmark investigates the impact of providing explicit reasoning tools, such as math for calculations or tools to synthesize collected information, particularly for tasks requiring multi-step analysis.
Enterprise Use Cases
AgentArch evaluates these configurations on two real-world enterprise use cases of varying complexity:
- Requesting Time Off (TO): A simpler, structured workflow for PTO eligibility verification and request processing. It involves 8 custom tools and 3 agents, testing basic multi-step reasoning.
- Customer Request Routing (CR): A more complex workflow mimicking intelligent customer service systems. It handles simple requests automatically and escalates complex issues, involving 31 custom tools and 9 agents, testing nuanced classification and escalation decisions.
The researchers deliberately constructed enterprise-realistic data, including lengthy knowledge base articles and complex JSON tool responses, to simulate the messy, unstructured information agents encounter in production environments.
Key Findings and Insights
The benchmark revealed several significant insights:
- Challenging Enterprise Workflows: Even state-of-the-art LLMs struggle with complex enterprise tasks. The simpler “Requesting Time Off” task achieved a peak success rate of 70.8%, while the complex “Customer Request Routing” task peaked at only 35.3%.
- Model-Specific Preferences: There is no one-size-fits-all optimal architecture. Models exhibited significant architectural preferences, and even within the same model, the best-performing configuration often differed between the simple and complex use cases.
- Function Calling vs. ReAct: Function calling generally outperformed ReAct across most models. Multi-agent ReAct consistently underperformed, and hallucinations were found almost exclusively under ReAct settings, suggesting that while models might be robust to hallucinations with trained function calling schemas, they are prone to them in ReAct.
- Thinking Tools: These tools consistently improved performance for non-reasoning models on simpler tasks, enabling more careful analysis. However, their impact was minimal on complex tasks and for models with strong inherent reasoning capabilities.
- Multi-Agent Systems for Final Decisions: While some models achieved higher overall scores in single-agent architectures, multi-agent systems were generally more effective at arriving at the correct final decision, even if they sometimes struggled with exact tool selection.
- Reliability Concerns: The reliability analysis, using pass@1 and pass^k scores, highlighted a fundamental gap between the promise of agentic LLMs and their real-world performance, with a low probability of consistently executing workflows correctly over multiple trials.
Also Read:
- Evaluating AI Agents: Introducing MCP-AgentBench for Real-World Tool Use
- Assessing Foundation Models for Planning Assistance
Recommendations for Enterprise AI Systems
Based on these findings, the researchers offer practical recommendations:
- Avoid Multi-Agent ReAct due to consistent underperformance and higher hallucination rates.
- Consider Multi-Agent Systems for scenarios where the correct final decision is paramount.
- Test against several use cases, as optimal architectures are task-dependent.
- Utilize thinking tools for non-reasoning models on tasks requiring calculations or data synthesis, but be mindful of potential latency.
- While larger models generally perform better, smaller models can achieve strong results on simpler tasks with the right architecture.
AgentArch provides crucial empirical data, enabling developers to make more informed decisions about architectural components and model selection when designing agentic AI systems for enterprise environments. It underscores the need for continued research to bridge the gap between current LLM capabilities and the high reliability demanded by real-world business applications.
