TLDR: DecentLLMs is a novel decentralized system for large language model (LLM) agents that addresses the limitations of leader-driven approaches in multi-agent collaboration. It enables worker agents to generate answers in parallel and evaluator agents to independently score and rank them using a Byzantine-robust aggregation algorithm (Geometric Median). This leaderless architecture ensures faster consensus, consistently selects higher-quality answers, and effectively tolerates malicious (Byzantine) agents, making it more resilient and efficient for LLM agent coordination, especially on open blockchain platforms.
Large Language Models (LLMs) have shown incredible capabilities across many tasks, from reasoning to translation. However, single LLM agents have their limitations, such as sometimes producing incorrect information (hallucinations) or being a single point of failure. To overcome these issues, researchers are increasingly looking into multi-agent systems, where multiple LLM agents work together to solve problems.
As these LLM agents are being deployed on open platforms like blockchains, it becomes crucial for these multi-agent systems to be able to tolerate malicious or ‘Byzantine’ agents. A Byzantine agent might deliberately try to sabotage the collaboration, leading to poor answers or delays.
Traditional Byzantine-robust multi-agent systems often rely on a leader-driven approach. In this setup, a designated leader agent coordinates the others to reach a consensus. While this works, it has significant drawbacks. Firstly, if the leader is malicious, the entire system can be vulnerable to targeted attacks, leading to repeated failures to reach consensus and costly restarts. Secondly, even if the leader isn’t malicious, their proposed answer might not be the best one available, but it could still be accepted if it gets enough votes, even when higher-quality alternatives exist.
To address these challenges, a new approach called DecentLLMs has been proposed. DecentLLMs introduces a novel decentralized consensus method for multi-agent LLM systems. Instead of relying on a single leader, DecentLLMs allows worker agents to generate answers concurrently. Then, evaluator agents independently score and rank these answers to select the best available one. This leaderless architecture offers several key advantages.
How DecentLLMs Works
DecentLLMs operates in a decentralized manner, ensuring all agents within their roles participate equally. It involves three main phases:
First, in the Answer Generation phase, worker agents independently create responses based on a user’s prompt. They work in parallel without communicating with each other, which helps improve response quality by reducing hallucinations and allows for horizontal scalability.
Next, the Answer Quality Evaluations phase begins. Evaluator agents receive all the answers from the workers and independently assess their quality. They use multiple criteria, such as checking for factual contradictions, fabrications, or inconsistencies with instructions or context. Each criterion is scored, and these scores are compiled into a vector for each answer.
Finally, the Byzantine-Robust Score Consensus phase takes place. Each evaluator agent broadcasts its evaluation scores to all other evaluators. To aggregate these scores reliably, even in the presence of malicious evaluators, DecentLLMs uses the Geometric Median (GM) algorithm. The GM algorithm is highly resilient to Byzantine attacks, meaning it can accurately find the true ‘median’ score even if some evaluators try to submit biased or false scores. After calculating a robust aggregated score for all answers, the evaluators identify the answer with the highest score as the final output. This answer is then sent back to the user, and a record of the process is stored on a blockchain for transparency and auditing.
Key Benefits and Performance
Experiments show that DecentLLMs significantly improves the quality of selected answers compared to traditional quorum-based voting methods. It achieved a 71% accuracy rate on a benchmark dataset, outperforming 2/3-quorum voting (64%) and majority quorum voting (50%). This is because DecentLLMs evaluates and ranks all worker answers, ensuring the highest-quality response is chosen, rather than just accepting a leader’s proposal that meets a minimum vote threshold.
Furthermore, DecentLLMs demonstrates a remarkably stable consensus latency. Unlike leader-driven approaches, where latency can increase dramatically with more Byzantine leaders due to repeated consensus rounds, DecentLLMs maintains a nearly constant latency. Its decentralized nature requires only a single round of communication among evaluators, making it more efficient in adversarial scenarios.
The system also exhibits strong Byzantine resilience. It successfully selects the correct answer even with a significant number of Byzantine evaluators, up to the theoretical fault tolerance limit of the Geometric Median algorithm. This highlights its robustness in challenging environments.
Also Read:
- FLock: Advancing Secure and Scalable LLM Training in Decentralized Networks
- Automating Gas Savings in Blockchain Contracts
Looking Ahead
DecentLLMs represents a significant step forward in decentralized multi-agent LLM systems, showcasing the benefits of a leaderless architecture for improved robustness and efficiency. Future work may explore integrating multi-round debates to further enhance worker response quality and reduce variance among evaluator scores. The researchers also envision using DecentLLMs as an AI-driven blockchain oracle, providing reliable and consistent outputs from non-deterministic LLM models for smart contract execution. You can read the full research paper here.
