DualTune: Enhancing On-Device AI Agents for Better Tool Calling

TLDR: DualTune is a novel methodology and inference framework designed to significantly improve the tool-calling capabilities of local, on-device Large Language Models (LLMs). It achieves this by disaggregating the tool-calling task into two distinct subtasks: tool selection and argument generation. Through “decoupled fine-tuning” with LoRA adapters, DualTune trains specialized components for each subtask. Additionally, “hierarchical orchestration” enables efficient handling of large toolsets by first selecting a toolset and then a specific tool within it. This approach leads to substantial accuracy improvements for local LLMs, often outperforming larger models, while maintaining privacy and reducing costs.

Large Language Models (LLMs) have transformed how we automate tasks, acting as central orchestrators for complex operations. However, deploying these powerful models directly on personal devices – for privacy, cost-effectiveness, and speed – has faced significant hurdles. Local LLMs often struggle with accurately selecting the right tool from a large set and generating the correct arguments for those tools, leading to performance gaps compared to their cloud-based counterparts.

A new research paper, DualTune: Decoupled Fine-Tuning for On-Device Agentic Systems, introduces a novel approach to overcome these limitations. Authored by Rohan Kadekodi, Zhan Jin, Keisuke Kamahori, Yile Gu, Sean Khatiri, Noah H. Bayindirli, Sergey Gorbunov, and Baris Kasikci, DualTune proposes a methodology that breaks down the intricate task of tool calling into two distinct, more manageable subtasks: tool selection and argument generation.

The Core Idea: Decoupled Fine-Tuning

The heart of DualTune lies in its “decoupled fine-tuning” strategy. Instead of training a single model to handle both tool selection and argument generation simultaneously, which are fundamentally different types of problems (classification vs. structured output generation), DualTune creates specialized components. It uses LoRA (Low-Rank Adaptation) fine-tuning to develop dedicated adapters:

• A common LoRA adapter for tool selection, responsible for identifying the most appropriate tool to use at each step.
• Separate, tool-specific LoRA adapters for argument generation, each designed to produce the precise and correctly formatted arguments for its particular tool.

This separation allows for more targeted training, using distinct loss masking for each subtask. For instance, the tool selection adapter is trained to focus only on predicting the tool name, while argument generation adapters concentrate solely on the parameters.

Hierarchical Orchestration for Scalability

One major challenge for local LLMs is managing a large number of tools, which can lead to excessively long context lengths and reduced accuracy. DualTune addresses this with “hierarchical orchestration,” a two-tiered selection process:

1. Toolset Selection: Initially, the base local LLM (without any fine-tuned adapters) performs a high-level routing task, identifying the most relevant toolset (e.g., ‘filesystem’ or ‘Notion’). This is a simpler decision that can be guided by a system prompt.
2. Tool Selection: Once a toolset is chosen, DualTune dynamically loads the specialized tool selection adapter for that specific toolset. This adapter then operates on a much smaller, more focused set of tools, significantly reducing complexity and improving accuracy.

This hierarchical approach keeps context lengths manageable, allowing the system to scale effectively to a greater number of tools without compromising performance.

The DualTune Inference Framework

DualTune integrates these techniques into a robust inference framework, implemented in Rust and powered by vLLM for efficient management of LoRA adapters. The workflow is streamlined:

1. The base LLM selects the appropriate toolset.
2. The relevant tool selection adapter is loaded, and an inference call identifies the desired tool.
3. The specific LoRA adapter for the chosen tool is dynamically loaded to generate its arguments.
4. The tool call is executed in a secure, containerized environment, and its output is added to the conversation history.
5. This process continues until the task is complete.

Impressive Performance Gains

Experiments on the MCP-Bench benchmark demonstrate DualTune’s effectiveness. The Qwen-2.5-7B model, when trained using decoupled fine-tuning (creating DualTuneModel-7B), showed a remarkable 46% improvement in tool calling accuracy over the base model. It consistently outperformed other local reasoning, non-reasoning, and fine-tuned models of similar size, and in many cases, even models twice its size.

Ablation studies further highlight the impact of each component: hierarchical orchestration alone improved accuracy, traditional fine-tuning added more, but decoupled fine-tuning delivered the most significant leap in performance, boosting accuracy from 16% to 61.5% in one test scenario.

Also Read:

• Tool-Light: A New Approach to Efficient AI Reasoning
• CoA-LoRA: Dynamic Adaptation for Quantized LLMs on Diverse Edge Devices

Looking Ahead

While adding new tools currently requires re-fine-tuning the common tool selector and creating a new adapter, the authors note that new tools are not frequently added to existing toolsets. Furthermore, DualTune provides an automated training pipeline that makes this process efficient, typically taking less than 10 hours. Future work aims to explore techniques like reinforcement learning to enable automatic adaptation to new tools during inference.

DualTune represents a significant step forward in making powerful, privacy-preserving agentic AI systems practical for deployment on everyday consumer devices, democratizing access to advanced LLM capabilities.