TLDR: A new research paper introduces a methodology to measure ‘epistemic diversity’ in Large Language Models (LLMs), revealing that while newer models show some improvement, they are generally less diverse than a web search. The study found that larger models tend to be less diverse, while Retrieval-Augmented Generation (RAG) positively impacts diversity. Critically, LLMs exhibit a significant knowledge gap, reflecting English-language information more than local language knowledge, highlighting a risk of knowledge collapse and erasure of non-English perspectives.
Large Language Models (LLMs) are increasingly becoming central to how we access information, from summarizing texts to answering complex queries. However, a recent study highlights a critical concern: these powerful AI tools tend to generate content that is remarkably similar in vocabulary, style, and even the underlying claims they make about the world. This phenomenon, termed ‘knowledge collapse,’ poses a risk of narrowing the range of accessible information over time, potentially dwindling our collective knowledge.
A new research paper, “Epistemic Diversity and Knowledge Collapse in Large Language Models,” by Dustin Wright, Sarah Masud, Jared Moore, Srishti Yadav, Maria Antoniak, Peter Ebert Christiansen, Chan Young Park, and Isabelle Augenstein, delves into this issue. The researchers introduce a novel methodology to measure ‘epistemic diversity,’ which refers to the variation in real-world claims found in LLM outputs. Their extensive study involved testing 27 different LLMs across 155 topics spanning 12 countries, using 200 prompt variations sourced from actual user conversations. For more details, you can read the full paper here.
Measuring the Breadth of Knowledge
To quantify epistemic diversity, the team developed a three-step process. First, they generated free-text responses from LLMs using natural writing assistance prompts. Second, these responses were broken down into individual, atomic claims. Finally, these claims were grouped into unique ‘meaning classes’ – clusters where all claims are semantically equivalent. The diversity of these meaning classes was then measured using Hill-Shannon diversity, a metric commonly used in ecology to assess species diversity.
Key Findings on Knowledge Diversity
The study revealed several significant insights into the state of LLM knowledge diversity:
- Diversity Over Time: While newer LLMs, particularly those released after March 2025 (like Gemma 3 and GPT-5), show an encouraging trend of increased epistemic diversity, nearly all models still generate less diverse claims compared to a basic web search. This suggests that relying solely on LLMs might offer a narrower perspective than traditional search engines.
- Impact of Model Size: Counter-intuitively, the research found that model size has a negative impact on epistemic diversity. Smaller models tend to generate more diverse knowledge than their larger counterparts. This challenges the assumption that bigger models always mean better or more comprehensive outputs.
- Retrieval-Augmented Generation (RAG) is Key: Retrieval-Augmented Generation (RAG), where LLMs retrieve information from external databases before generating a response, was found to have a statistically significant positive impact on epistemic diversity. This highlights RAG’s potential in preventing knowledge collapse, provided that the RAG databases themselves remain diverse and are not contaminated by an overabundance of LLM-generated content.
- Cultural Context and Representation Gaps: The study also examined how cultural context affects diversity. While instruction-fine-tuned models showed similar diversity across countries, RAG had an uneven impact, with countries like the USA, India, Russia, France, and China seeing more benefit. This is likely due to the search data used for RAG being performed in the US region, potentially under-representing information from other cultures. Furthermore, when comparing LLM-generated claims to Wikipedia content in both English and local languages, the study found a significant knowledge gap: LLM outputs reflected English language knowledge more than local language knowledge for many countries. This points to a critical need to improve the representation of local knowledge in LLM outputs to avoid knowledge erasure.
Also Read:
- Unpacking Historical Bias in AI: A Cross-Lingual Look at LLMs and Romanian History
- AI’s Mirror Test: Large Language Models Struggle to Recognize Their Own Creations
Implications for the Future of AI
The findings underscore that while LLMs are improving, their epistemic diversity remains low compared to traditional search methods. To mitigate the risk of knowledge collapse, the researchers recommend prioritizing RAG and using smaller models where epistemic diversity is crucial. They also emphasize the importance of maintaining diverse RAG databases and actively working to incorporate local, non-English knowledge into LLM training and retrieval processes. This research provides a valuable methodology for ongoing assessment of how much, which, and whose knowledge LLMs are truly representing.
