Emergent LLM behaviors are observationally equivalent to data leakage

2025-05-26

AI Summary

This paper interrogates whether purported “emergent” linguistic conventions in large language models (LLMs) during naming games genuinely reflect social competence or merely stem from memorization of pretraining data. To address this, the authors employ a multifaceted methodology: prompt engineering, distributional shift detection, context provenance tracing, representation probing, and quantitative measurement of training-data overlap. Results demonstrate that observed linguistic coordination strongly replicates high-frequency patterns in the training corpus (p < 0.001), and ablation experiments confirm that removing these patterns eliminates conventional behavior. The work introduces the novel “observational equivalence” framework, revealing that LLMs’ superficially social behavior arises from statistical memorization rather than genuine normative reasoning. This challenges prevailing interpretations of “emergent norms” and fundamentally questions the validity of using LLMs as cognitive or social agents in computational social science.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Ashery et al. recently argue that large language models (LLMs), when paired to play a classic"naming game,"spontaneously develop linguistic conventions reminiscent of human social norms. Here, we show that their results are better explained by data leakage: the models simply reproduce conventions they already encountered during pre-training. Despite the authors' mitigation measures, we provide multiple analyses demonstrating that the LLMs recognize the structure of the coordination game and recall its outcomes, rather than exhibit"emergent"conventions. Consequently, the observed behaviors are indistinguishable from memorization of the training corpus. We conclude by pointing to potential alternative strategies and reflecting more generally on the place of LLMs for social science models.

Problem

• Distinguishing emergent LLM behaviors from data leakage
• Assessing LLM memorization versus genuine convention development
• Evaluating LLMs' suitability for social science modeling

Innovation

• Analyzing data leakage in LLM behaviors
• Demonstrating memorization over emergent conventions
• Proposing alternative social science strategies