Computational Turing Test Reveals Systematic Differences Between Human and AI Language

2025-11-06

AI Summary

This study addresses the lack of empirical validation for anthropomorphic behavior simulation in large language models (LLMs). We propose the first automated Computational Turing Test framework integrating semantic similarity, detectability analysis, and interpretable linguistic features—including stylistic markers and topical patterns. Systematically evaluating nine open-source LLMs on multi-platform real-world social media data, we compare calibration strategies including instruction tuning, prompt engineering, and retrieval-augmented generation. Results reveal that even extensively calibrated models significantly deviate from human text—particularly in affective tone; instruction tuning degrades anthropomorphism; scaling parameters yields no anthropomorphic improvement; and anthropomorphic optimization often compromises semantic fidelity. Crucially, this work identifies a fundamental trade-off between semantic consistency and human-likeness—the first such empirical demonstration—and establishes a reproducible, interpretable benchmarking paradigm for LLM-based social simulation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) are increasingly used in the social sciences to simulate human behavior, based on the assumption that they can generate realistic, human-like text. Yet this assumption remains largely untested. Existing validation efforts rely heavily on human-judgment-based evaluations -- testing whether humans can distinguish AI from human output -- despite evidence that such judgments are blunt and unreliable. As a result, the field lacks robust tools for assessing the realism of LLM-generated text or for calibrating models to real-world data. This paper makes two contributions. First, we introduce a computational Turing test: a validation framework that integrates aggregate metrics (BERT-based detectability and semantic similarity) with interpretable linguistic features (stylistic markers and topical patterns) to assess how closely LLMs approximate human language within a given dataset. Second, we systematically compare nine open-weight LLMs across five calibration strategies -- including fine-tuning, stylistic prompting, and context retrieval -- benchmarking their ability to reproduce user interactions on X (formerly Twitter), Bluesky, and Reddit. Our findings challenge core assumptions in the literature. Even after calibration, LLM outputs remain clearly distinguishable from human text, particularly in affective tone and emotional expression. Instruction-tuned models underperform their base counterparts, and scaling up model size does not enhance human-likeness. Crucially, we identify a trade-off: optimizing for human-likeness often comes at the cost of semantic fidelity, and vice versa. These results provide a much-needed scalable framework for validation and calibration in LLM simulations -- and offer a cautionary note about their current limitations in capturing human communication.

Problem

• Assessing realism of LLM-generated text through computational validation methods
• Comparing AI and human language differences across multiple platforms and models
• Identifying trade-offs between human-likeness and semantic fidelity in LLMs

Innovation

• Computational Turing test with aggregate metrics and linguistic features
• Systematic comparison of nine LLMs across five calibration strategies
• Identified trade-off between human-likeness and semantic fidelity