TLDR: The research paper introduces the Agentic Meta-Orchestrator (AMO), a system designed to efficiently manage and coordinate multiple AI agents within copilot services like Microsoft 365 E-Commerce Copilot and Code Compliance Copilot. AMO addresses challenges in task distribution, memory-efficient inference, and dynamic inference planning by using a multi-level learning-to-rank model for agent selection, LoRA arms for shared LLM memory across tasks, and a meta-learning decision tree for optimal agent sequencing. This approach enables copilots to provide more accurate, up-to-date, and customized responses, outperforming traditional methods and existing agentic frameworks.
Microsoft Copilot suites are designed to be a central hub for various AI agents, each specialized in handling different tasks, from assisting customers with purchases to detecting code vulnerabilities. As the number of these agents grows, a robust system is needed to effectively distribute tasks from user prompts to the correct agents. This challenge is addressed by the Agentic Meta-Orchestrator (AMO).
The AMO framework, proposed by Xiaofeng Zhu and Yunshen Zhou from Microsoft Corporation, offers a sophisticated solution for managing multiple tasks and scalable agents within copilot services. It is capable of providing both natural language and action-based responses, significantly enhancing the functionality and efficiency of AI copilots.
Addressing Key Challenges in Copilot Orchestration
The researchers identified three primary challenges in developing scalable and effective copilot services:
- Effective Task Distribution: Decomposing user prompts and assigning them to the right agents as the agent ecosystem expands.
- Efficient Deployment: Optimizing the deployment of foundational language models and agents.
- Inference Planning: Deciding which agents to use and in what sequence, and how information can be shared between them.
To tackle the first challenge, AMO employs a trained multi-level rating learning-to-rank model. Unlike traditional multi-class text classification, which struggles with scalability when new agents are added, this approach treats agent selection as a hierarchical text classification problem. It uses natural language descriptions of agent labels as ‘candidate documents’ and predicts a relatedness score, allowing for seamless integration of new agents without retraining the entire model. This method also enables the selection of top-k relevant agents based on user prompts, using a ‘Separator Class’ to define the relevance cutoff.
For efficient deployment, AMO introduces a memory-efficient inference framework called LoRA arms. This innovative approach allows multiple LoRA (Low-Rank Adaptation) arms, each trained for a specific task, to share the same base Large Language Model (LLM) memory during inference. This significantly reduces memory consumption compared to hosting multiple separate LLMs, making the system more scalable and flexible for complex AI systems like copilots.
The third challenge, inference planning, is addressed by a novel meta-learning decision tree model. Instead of relying on predefined workflows or cognitive heuristics, this model learns the optimal combination and ordering of agents for different user prompts. The training inputs include user prompts, their end-to-end copilot responses, and various agent-specific models. This meta-learner ensures that the copilot can dynamically choose the best inference strategy, such as whether to use a product recognition agent, a database agent, or a combination thereof.
Also Read:
- AgentArcEval: A New Approach to Assessing Foundation Model Agent Design
- DeepAgent: Advancing AI with Autonomous Reasoning and Dynamic Tool Use
Real-World Applications and Performance
The effectiveness of AMO has been demonstrated through two production use cases at Microsoft:
- Microsoft 365 (M365) E-Commerce Copilot: This copilot helps external customers with product purchases, providing up-to-date product information and connecting to agents like relational databases and human customer support. AMO enables it to offer customized responses based on context, geographic location, and multi-turn conversations, a capability that standalone LLMs like ChatGPT often lack.
- Code Compliance Copilot: This copilot scans internal DevOps code to detect compliance issues in pull requests, including known and new GB compliance issues. AMO’s hierarchical classification capabilities are crucial here for identifying different levels of compliance problems.
Experimental results show that AMO significantly outperforms standalone LLMs (Fine-tuned Phi-3.5, ChatGPT-4o) and other agentic workflow engines like AutoGen in terms of ROUGE-L, BERTScore, and F-1 measures for agent orchestration and product recognition tasks. The agentic orchestrator’s performance remains stable even as the number of agents grows, a key advantage over traditional BERT models.
The research paper highlights that AMO’s approach of treating multi-label or hierarchical text classification as a multi-level rating learning-to-rank task is more effective than generation tasks for these specific applications. The framework’s ability to handle growing agents, optimize memory usage, and dynamically plan inference paths makes it a powerful advancement for multi-task AI copilots. For more technical details, you can refer to the full research paper here.
