TLDR: The research introduces In-N-Out, the first expert-annotated dataset of parameter-level API graphs, designed to help LLM-based tool agents understand and utilize external APIs more effectively. Tool agents often struggle with complex multi-tool queries requiring chained API calls due to difficulty in identifying API dependencies from documentation. In-N-Out converts API documentation into a structured graph, explicitly mapping input-output parameter relationships. Fine-tuning LLMs on In-N-Out significantly improves their ability to construct these graphs and enhances performance in tool retrieval and multi-tool query generation, demonstrating that explicit API graphs are crucial for advancing tool agent capabilities.
Large Language Models (LLMs) are becoming increasingly capable, leading to the rise of “tool agents” – systems that can interact with external APIs to perform real-world tasks. These agents can retrieve real-time information, access databases, and operate services, going beyond the limitations of pre-trained knowledge. However, as tasks become more complex, requiring multiple API calls in a specific sequence (known as multi-tool queries), these agents often struggle. The core challenge lies in identifying the correct APIs and understanding their interdependencies to plan valid call sequences.
Imagine an agent trying to fulfill a request like “Follow all the EDM artists on Spotify that have at least 1K followers.” This seemingly simple task requires a series of steps: first, logging in to obtain an access token, then searching for artists based on genre and follower count, and finally, using the obtained artist IDs and access token to follow them. Without a clear understanding of how the output of one API (like an access token or artist ID) feeds into the input of another, agents frequently fail to chain these operations correctly.
Current approaches often involve training agents on vast datasets of multi-tool queries or refining API documentation. However, LLMs find it difficult to grasp parameter-level dependencies from the often noisy and inconsistent real-world API documentation. This is where the new research introduces a novel solution: converting API documentation into a structured API graph.
Introducing In-N-Out: A Dataset for API Graphs
Researchers have developed In-N-Out, the first expert-annotated dataset of API graphs. This dataset is built from two real-world API benchmarks, AppWorld and NESTful, encompassing 550 APIs and over 30,000 parameter-level connections. In-N-Out represents both APIs and their individual parameters as nodes in a graph. Directed edges then indicate when an output from one API can serve as a valid input to another, effectively mapping out the flow of information.
The creation of In-N-Out involved a meticulous multi-stage pipeline. First, API documentation was refined, flattening complex object structures into individual parameters and pruning auxiliary values to reduce noise. Then, a candidate pair filtering process was applied, using rule-based, semantic, and context-aware methods to identify plausible connections between output and input parameters. Finally, human experts, experienced software developers, independently annotated these candidate pairs based on two criteria: data compatibility (whether an output can validly serve as an input) and naturalness (whether the connection reflects realistic tool usage). This led to the classification of connections into ‘strong-edges’ (compatible and natural), ‘weak-edges’ (conditional and natural), and ‘non-edges’.
Significant Performance Improvements
The impact of In-N-Out is substantial. Experiments showed that fine-tuning LLMs on this dataset significantly improves their ability to construct parameter-level API graphs from documentation, even for APIs they haven’t seen before. Zero-shot models (without fine-tuning) performed poorly, highlighting the difficulty of this task for LLMs alone. However, models fine-tuned on In-N-Out achieved much higher accuracy, demonstrating that the dataset helps models learn to comprehend API documentation and the intricate relationships between parameters.
Furthermore, using In-N-Out dramatically boosted performance in two key areas for tool agents: tool retrieval and multi-tool query generation. In tool retrieval, where an agent needs to find a prerequisite API to supply a missing input, API graphs significantly improved accuracy. For instance, in the larger AppWorld dataset, the average rank of the correct API improved from 19.3 without a graph to 4.5 with the gold graph. Even graphs automatically generated by fine-tuned models closed much of this performance gap, showing the practical utility of learned graph structures.
For multi-tool query generation, which is crucial for training and benchmarking agents, In-N-Out also proved invaluable. By helping models select subsets of APIs that can be composed into valid sequences, access to graph information significantly improved the precision of identifying API subsets that match predefined structural patterns (like chains, forks, or colliders). This is particularly helpful in complex environments with many APIs and cross-domain connections.
Also Read:
- AnchorRAG: A Multi-Agent Framework for Enhanced Open-World Question Answering with Knowledge Graphs
- Enhancing Language Models for Graph Tasks with Targeted Context
The Future of Tool Agents
The In-N-Out dataset and the findings from this research highlight the immense promise of using explicit API graphs for tool agents. By providing a structured way to understand API dependencies, these graphs enable LLMs to move beyond superficial heuristics and reason more effectively about how APIs can be composed to solve complex tasks. The research paper, titled “In-N-Out: A Parameter-Level API Graph Dataset for Tool Agents,” is available for further reading here.
While In-N-Out currently covers two benchmark datasets, future work aims to expand its coverage to broader domains and explore semi-automated approaches for graph construction to streamline the process while maintaining quality. This will further enhance the development of robust and capable tool agents.
