TLDR: A new research paper introduces SMACS, a Scalable Multi-Agent Collaboration System that enables multiple open-source Large Language Models (LLMs) to work together. By using a retrieval-based selection process to pick expert LLMs and an exploration-exploitation strategy to refine answers, SMACS has shown to significantly outperform leading closed-source LLMs like GPT-4.1 and Claude-3.7-Sonnet across various benchmarks, demonstrating the power and scalability of open-source collaboration.
In the rapidly evolving landscape of Large Language Models (LLMs), a significant question has emerged: can a collaborative system of multiple open-source LLMs collectively surpass the performance of powerful closed-source models? A recent research paper, titled “Open-Source LLMs Collaboration Beats Closed-Source LLMs: A Scalable Multi-Agent System,” by Shengji Tang, Jianjian Cao, Weihao Lin, Jiale Hong, Bo Zhang, Shuyue Hu, Lei Bai, Tao Chen, Wanli Ouyang, and Peng Ye, delves into this very question, proposing an innovative framework called SMACS.
The core challenge addressed by this research is the increasing computational overhead and diminishing returns from optimizing single, massive LLMs. Concurrently, a diverse array of open-source LLMs, each with specialized skills due to varied training data, has become available. This diversity presents a unique opportunity for collaboration, potentially pushing the boundaries of AI performance.
Introducing SMACS: A Collaborative Breakthrough
SMACS, which stands for Scalable Multi-Agent Collaboration System, is designed to harness the collective intelligence of multiple open-source LLMs. It operates in two main stages to achieve high performance and adaptability:
First, there’s the **Retrieval-based Prior Selection (RPS)**. Imagine a vast library of questions and how different LLMs performed on them in the past. When SMACS receives a new question, it intelligently sifts through this historical data to find similar questions. Based on which LLMs performed best on those similar questions, SMACS selects a group of “expert” LLMs that are most likely to provide accurate answers for the current query. This process ensures that only the most relevant and capable models are engaged, saving computational resources and improving accuracy.
Second, SMACS employs an **Exploration–Exploitation-Driven Posterior Enhancement (EPE)**. Once the selected expert LLMs generate their responses, SMACS doesn’t just pick one. Instead, it strategically drops some less promising responses (based on their initial scores) to create multiple diverse subsets of answers. Each subset is then aggregated by a powerful LLM (the “aggregator”) to form several candidate final responses. To pick the best one, SMACS uses a clever “hybrid score” that considers both how similar the aggregated response is to others and its overall coherence (perplexity). This two-pronged approach allows SMACS to explore a wide range of potential answers and then exploit the best one, leading to higher quality outputs.
Also Read:
- Decentralized LLM Agent Coordination: A Robust Approach to Multi-Agent Systems
- Unlocking Advanced AI Reasoning with Adaptive Multi-Agent Systems
Remarkable Performance and Scalability
The experimental results of SMACS are truly impressive. By integrating fifteen mid-sized open-source LLMs, SMACS demonstrated superior performance across eight mainstream benchmarks, including mathematical problem-solving, complex reasoning, instruction following, and code generation. It significantly outperformed leading closed-source LLMs from 2025, such as Claude-3.7-Sonnet, GPT-4.1, and GPT-o3-mini, with substantial accuracy gains. For instance, SMACS achieved an average accuracy of 76.78% across benchmarks, a notable increase over both closed-source (65.66%) and open-source (59.66%) baselines.
Beyond just outperforming individual models, SMACS even exceeded the average best results from both open-source and closed-source LLMs, effectively pushing the “upper bound of intelligence.” The research also highlights SMACS’s excellent scalability, showing that its performance consistently improves as more LLMs are incorporated into the system, without significant saturation.
This research underscores the immense potential of collaborative open-source LLM systems. It suggests a future where the combined strengths of diverse open models can not only rival but surpass the capabilities of proprietary, closed-source alternatives. The code for SMACS is planned to be released, fostering further innovation in the field. You can find more details in the full research paper available at arXiv:2507.14200.
