Epistemic Diversity and Knowledge Collapse in Large Language Models

2025-10-05

AI Summary

Large language models (LLMs) exhibit pronounced cognitive homogenization in generated content, exacerbating knowledge collapse—the progressive narrowing of accessible knowledge breadth over time. Prior work relies on closed-domain evaluations or vague semantic metrics, lacking longitudinal and cross-cultural perspectives. Method: We propose the first quantitative framework for cognitive diversity grounded in real user prompts and multilingual, geopolitically diverse topics. It integrates retrieval-augmented generation (RAG), multi-turn dialogue variants, cross-lingual topic sampling, and comparative analysis against authoritative knowledge sources (e.g., Wikipedia). Contribution/Results: Empirical evaluation reveals that mainstream LLMs consistently yield lower cognitive diversity than baseline web search; scaling model parameters further intensifies homogenization; RAG mitigates this effect only partially, with efficacy strongly moderated by cultural context. This study is the first to systematically expose structural biases between model scale and cultural representativeness, establishing a novel paradigm for assessing and improving AI knowledge health.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) tend to generate lexically, semantically, and stylistically homogenous texts. This poses a risk of knowledge collapse, where homogenous LLMs mediate a shrinking in the range of accessible information over time. Existing works on homogenization are limited by a focus on closed-ended multiple-choice setups or fuzzy semantic features, and do not look at trends across time and cultural contexts. To overcome this, we present a new methodology to measure epistemic diversity, i.e., variation in real-world claims in LLM outputs, which we use to perform a broad empirical study of LLM knowledge collapse. We test 27 LLMs, 155 topics covering 12 countries, and 200 prompt variations sourced from real user chats. For the topics in our study, we show that while newer models tend to generate more diverse claims, nearly all models are less epistemically diverse than a basic web search. We find that model size has a negative impact on epistemic diversity, while retrieval-augmented generation (RAG) has a positive impact, though the improvement from RAG varies by the cultural context. Finally, compared to a traditional knowledge source (Wikipedia), we find that country-specific claims reflect the English language more than the local one, highlighting a gap in epistemic representation

Problem

• LLMs produce homogeneous texts risking knowledge collapse over time
• Existing methods fail to capture temporal and cultural diversity trends
• Models show reduced epistemic diversity compared to web searches

Innovation

• Developed methodology to measure epistemic diversity
• Tested 27 LLMs across 155 topics globally
• Found RAG improves diversity with cultural variations