TLDR: A new research paper introduces CORE, a metric to quantify linguistic diversity and quality in multi-agent LLM interactions under game-theoretic conditions (cooperative, competitive, neutral). The study found that neutral interactions are the most linguistically diverse, while cooperative settings lead to more repetition and vocabulary expansion, and competitive settings result in less repetition and constrained vocabularies. CORE provides a direct evaluation of how social incentives influence language adaptation and can identify mode collapse in multi-agent LLM systems.
Large Language Models (LLMs) are increasingly interacting with each other in multi-agent systems, revealing fascinating new capabilities. However, understanding and quantifying the quality and diversity of language used in these interactions, especially under different social pressures, has been a significant challenge. A new research paper introduces a novel metric called CORE, the Conversational Robustness Evaluation Score, designed to address this very issue.
The CORE metric provides a direct way to measure the effectiveness and quality of language within multi-agent LLM systems. It achieves this by integrating several key linguistic aspects: cluster entropy (how varied the conversational topics or styles are), lexical repetition (how often words are repeated), and semantic similarity (how similar the meanings of consecutive utterances are). By combining these measures, CORE offers a comprehensive view of dialog quality.
To ground their analysis, the researchers applied CORE to pairwise LLM dialogs across three distinct game-theoretic settings: competitive, cooperative, and neutral. They also incorporated well-established linguistic laws, Zipf’s Law and Heaps’ Law, which describe word frequency distributions and vocabulary growth, respectively. Zipf’s Law suggests that a few words are used very frequently, while Heaps’ Law models how vocabulary size grows with the length of a text.
The findings from this study offer compelling insights into how social incentives influence language adaptation in LLMs. In cooperative settings, where agents work together towards a shared goal, the study observed both steeper Zipf distributions and higher Heap exponents. This indicates that while agents expand their vocabulary, they also exhibit more repetition, likely converging on shared terminology to achieve their common objective. For example, in a cooperative puzzle-solving scenario, agents might frequently use words like “puzzle,” “solve,” and “together.”
Conversely, competitive interactions, where agents have adversarial objectives, displayed lower Zipf and Heaps exponents. This suggests less repetition and more constrained vocabularies, as agents might be more focused on strategic, concise communication rather than broad exploration of language. Neutral settings, where agents engage in open-ended conversation without specific agendas, consistently showed the highest CORE values, indicating the most lexically diverse and varied interactions.
The research also delved into behavioral metrics, revealing that competitive dialogs exhibited significantly higher toxicity scores. In contrast, neutral settings showed lower repetition rates and more varied interactions, aligning with the higher CORE scores observed in these conditions. The study utilized a range of open-source LLMs, including Llama-3.1, Gemma, Qwen, and Mistral, across thousands of interactions to ensure robust evaluation.
Also Read:
- Unpacking LLM Strategic Reasoning: A New Benchmark Approach
- Unpacking How Large Language Models Interact with External Tools: A New Study’s Insights
The CORE metric serves as a robust diagnostic tool for measuring linguistic robustness in multi-agent LLM systems. It highlights how LLMs adapt their language in response to different social pressures, sometimes leading to repetitive or semantically stagnant communication even without explicit multi-agent training. This work paves the way for better understanding and developing more sophisticated and diverse communication in future AI systems. For more detailed information, you can refer to the full research paper available here.
