Wikipedia in the Era of LLMs: Evolution and Risks

2025-03-04

AI Summary

This study investigates the reverse impact of large language models (LLMs) on the Wikipedia content ecosystem and its implications for NLP evaluation integrity. We employ a multi-method empirical framework—including large-scale statistical analysis of real-world data, LLM-generated content detection, machine translation quality assessment, RAG performance benchmarking, and reverse-generation simulation modeling. Our analysis quantifies, for the first time, that 1–2% of Wikipedia articles are contaminated by LLM-generated content. We identify two novel risk paradigms: (1) LLM contamination artificially inflates scores on mainstream machine translation benchmarks, inducing systematic evaluation bias; and (2) contaminated content degrades RAG retrieval relevance and answer accuracy. These findings demonstrate that LLMs are silently eroding the credibility of open knowledge repositories and expose systemic flaws in current NLP evaluation practices. The work provides critical empirical evidence to inform knowledge-base governance and the development of robust, trustworthy AI evaluation frameworks.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

In this paper, we present a thorough analysis of the impact of Large Language Models (LLMs) on Wikipedia, examining the evolution of Wikipedia through existing data and using simulations to explore potential risks. We begin by analyzing page views and article content to study Wikipedia's recent changes and assess the impact of LLMs. Subsequently, we evaluate how LLMs affect various Natural Language Processing (NLP) tasks related to Wikipedia, including machine translation and retrieval-augmented generation (RAG). Our findings and simulation results reveal that Wikipedia articles have been influenced by LLMs, with an impact of approximately 1%-2% in certain categories. If the machine translation benchmark based on Wikipedia is influenced by LLMs, the scores of the models may become inflated, and the comparative results among models might shift as well. Moreover, the effectiveness of RAG might decrease if the knowledge base becomes polluted by LLM-generated content. While LLMs have not yet fully changed Wikipedia's language and knowledge structures, we believe that our empirical findings signal the need for careful consideration of potential future risks.

Problem

• Analyze LLMs' impact on Wikipedia evolution and risks.
• Evaluate LLMs' effects on NLP tasks like translation and RAG.
• Assess potential risks of LLM-generated content on Wikipedia.

Innovation

• Analyzed Wikipedia changes using LLM impact data
• Simulated risks of LLMs on NLP tasks
• Assessed LLM influence on Wikipedia content quality