TLDR: The ACE (Automated Creation and Enrichment) framework is an innovative solution designed to make enterprise APIs more accessible and usable for Large Language Model (LLM) agents. It addresses common challenges like poor documentation, complex schemas, and large API sets by automatically generating enriched tool specifications with detailed parameter descriptions and examples. Additionally, ACE incorporates a dynamic shortlisting mechanism that filters relevant tools at runtime, reducing prompt complexity and improving scalability. This framework significantly enhances LLM agents’ ability to accurately select and invoke APIs, as demonstrated by its effectiveness on both proprietary and open-source APIs and its deployment in IBM Watsonx Orchestrate.
Large Language Models (LLMs) are becoming increasingly capable, allowing for the creation of sophisticated AI agents that can perform complex reasoning and interact with external tools. However, integrating these agents with enterprise Application Programming Interfaces (APIs) often presents significant hurdles. Issues like inadequate documentation, intricate input/output structures, and the sheer volume of available operations can make it difficult for LLMs to select the right tool and form accurate requests, potentially reducing performance by up to 25%.
Introducing ACE: A Solution for Enterprise API Integration
To tackle these challenges, researchers from IBM Research, India, have developed a novel framework called ACE: Automated Creation and Enrichment. ACE is designed to transform complex enterprise APIs into tools that are easily understood and invoked by LLM-based agents. It’s an end-to-end system that automates the creation, enrichment, and dynamic selection of these API tools.
How ACE Enhances API Tools
ACE focuses on two primary mechanisms to improve LLM agent performance:
-
Enriched Tool Specifications: ACE automatically generates detailed descriptions for API parameters and provides illustrative examples. This rich metadata helps LLMs make better decisions when selecting tools and constructing the correct input payloads.
-
Dynamic Shortlisting: For environments with many APIs, ACE incorporates a mechanism to filter and present only the most relevant tools to the LLM at runtime. This reduces the complexity of the prompt the LLM needs to process, maintaining scalability without sacrificing accuracy.
Addressing Key Challenges
The framework directly addresses several pain points in integrating enterprise APIs with LLM agents:
-
Manual Conversion: Traditionally, converting API specifications into agent-compatible tools is a manual, time-consuming, and error-prone process.
-
Poor Documentation: Enterprise APIs often lack the structured, detailed, and user-friendly documentation that LLMs need to understand their purpose and usage.
-
Scalability Issues: APIs with hundreds or thousands of operations (like Jira’s 900+ API operations) can overwhelm an LLM’s context window, making it impractical to consider all available tools.
ACE automates the generation of clear tool-level descriptions, parameter documentation with type information, and example values. These examples act as ‘few-shot demonstrations,’ guiding the LLM to form valid input payloads and reducing errors.
The ACE Framework in Action
The ACE framework operates in several stages:
-
OAS Metadata Enrichment: It analyzes existing API specifications (OpenAPI Specification or OAS) to infer implicit semantic information from elements like endpoint paths, HTTP methods, and parameter definitions. Using LLMs, it generates missing or unclear descriptions for API methods and parameters, and creates meaningful parameter examples consistent with any defined constraints.
-
API Tool Creation: ACE parses the OAS to generate framework-specific Python tools. It uses the operation ID as the function name, extracts parameters for function arguments, handles request headers, and constructs a comprehensive docstring. This docstring includes the enriched endpoint and parameter descriptions, along with an illustrative input example to guide the LLM.
-
Tool Shortlisting: For large tool catalogs, ACE employs a Retrieval-Augmented Generation (RAG)-based approach. It embeds user queries and tool descriptions into a shared vector space and uses semantic search to identify the top-k most relevant tools. This dynamic shortlisting ensures that the agent only considers a manageable and highly pertinent subset of tools.
Also Read:
- Evaluating AI Agent Architectures for Enterprise Workflows
- The True Cost of AI in Software Engineering: Introducing SWE-Effi
Evaluation and Impact
The researchers validated ACE on both proprietary (Salesloft) and open-source (Kubernetes) enterprise APIs. Their experiments showed that enriched metadata significantly improves tool calling performance, especially when detailed parameter descriptions and examples are included (referred to as Enrich-3). The benefits were observed to be model-dependent, with medium-sized models like Llama-70B showing consistent improvements, while smaller models like Granite-8B could be sensitive to excessive verbosity.
Crucially, enrichment boosted tool shortlisting accuracy, particularly when the agent needed to select from a smaller number of top-ranked tools. This capability helps LLM agents effectively manage large tool catalogs without being constrained by context length.
The ACE framework is already deployed as part of the alpha release of IBM Watsonx Orchestrate Agent Development Kit (ADK). Internal experiments demonstrated a 27% improvement in tool selection and invocation accuracy when using ACE-enriched metadata compared to minimal metadata, achieving performance comparable to manually authored metadata. This indicates a significant step towards making enterprise APIs more accessible and effective for LLM-based agents.
For more details, you can read the full research paper here.
