TLDR: The MoMA (Mixture of Models and Agents) framework addresses the challenge of efficiently routing diverse user queries to the most appropriate AI models or agents. It combines LLM-based and agent-based routing, using a two-layer system for agents and a performance-cost optimization for LLMs. MoMA learns from a large dataset to understand model capabilities, dynamically selecting the best execution unit to balance performance and cost. Experimental results show MoMA significantly reduces costs while maintaining high performance, demonstrating its adaptability and scalability in real-world AI applications.
In the rapidly expanding world of artificial intelligence, large language models (LLMs) and specialized AI agents are becoming increasingly powerful and diverse. From general-purpose LLMs like GPT-5 to domain-specific tools for medical applications or code generation, the AI ecosystem is rich with capabilities. However, this diversity also brings a significant challenge: how do we efficiently and accurately direct a user’s query to the most suitable AI tool, balancing both performance and cost?
Researchers from the JIUTIAN Team at China Mobile Research Institute have proposed an innovative solution called MoMA, which stands for Mixture of Models and Agents. This framework introduces a generalized routing system designed to orchestrate both LLMs and AI agents for adaptive and efficient inference. MoMA aims to understand the unique strengths of various models and agents, allowing it to precisely recognize user intent and route queries to the optimal execution unit.
Understanding the MoMA Framework
The core idea behind MoMA is to act as a smart traffic controller for AI queries. When a user submits a request, MoMA first determines if a specialized AI agent can handle it. Agents are often highly deterministic and efficient for specific tasks. If an agent is suitable, MoMA selects the best one. If not, the system intelligently falls back to an LLM, choosing the most appropriate one from a pool of diverse models.
Smart Agent Selection
MoMA’s agent routing works in two layers, much like a “divide and conquer” strategy. The first layer performs a broad classification, grouping agents into high-level categories such as “Image,” “Travel,” or “Meeting.” This helps narrow down the possibilities. The second layer then uses a “context-aware finite state machine” for a more precise selection. This state machine dynamically loads detailed descriptions of candidate agents, ensuring that the final choice is made from a relevant set. A clever “token logits masking” strategy is also employed, which prevents the system from trying to use agents that are unavailable or unsuitable, significantly improving accuracy without adding extra cost.
Optimizing LLM Choices
When a query is routed to an LLM, MoMA doesn’t just pick the biggest or most popular model. Instead, it evaluates the performance of various LLMs for that specific query and considers their operational costs. It then constructs a “Pareto frontier” curve, which helps visualize the trade-off between performance and cost. Using an algorithm called TOPSIS, MoMA identifies the LLM that offers the best balance, ensuring optimal efficiency and effectiveness. Users can even specify preferences, such as prioritizing performance, cost, or allowing MoMA to find an automatic balance.
Training and Validation
To achieve this intelligent routing, MoMA was trained on a massive dataset of approximately 2.25 million instances. This dataset was carefully constructed to cover a wide range of domains, task complexities, and difficulty levels. The training process involved evaluating how different LLMs performed on various tasks, using an “LLM-as-a-judge” approach to compare model capabilities. This meticulous training allows MoMA to accurately characterize the strengths and weaknesses of each LLM and agent.
Also Read:
- Optimizing Large Language Model Routing for Cost and Performance
- Tree of Agents: A Multi-Perspective Approach to Long-Context Understanding in LLMs
Real-World Impact
Experimental results have shown that MoMA offers superior cost-efficiency and scalability compared to existing methods. In performance-priority scenarios, MoMA achieved comparable or even better performance than the best single LLM, while significantly reducing costs by over 30%. Its automatic routing strategy also delivered strong performance at a much lower cost, demonstrating an optimal balance. The framework has already been successfully deployed with a variety of high-quality models and over 20 expert agents, handling real-world applications in programming, mathematics, translation, and healthcare.
By intelligently directing queries to the most appropriate AI model or agent, MoMA not only improves the efficiency of task execution but also contributes to a more open and compatible AI ecosystem, where even smaller, specialized models can be effectively utilized. For more technical details, you can refer to the full research paper: Towards Generalized Routing: Model and Agent Orchestration for Adaptive and Efficient Inference.
