Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AI

2025-02-17

AI Summary

This study investigates human ability to distinguish and prefer AI-generated versus human-written text across multilingual and multidomain settings. We conduct a large-scale, manually annotated evaluation involving 16 datasets, 9 languages, and 9 thematic domains, establishing a multilingual detectability assessment framework. Our results reveal— for the first time—that cross-lingual human detection accuracy reaches 87.6%, substantially challenging the prevailing assumption of indistinguishability between AI- and human-authored texts. We identify textual specificity, cultural adaptability, and expressive diversity as key discriminative dimensions. Furthermore, we propose a cognition-informed controllable generation method that significantly enhances anthropomorphism in over 50% of cases. Crucially, we demonstrate that human preference does not inherently favor human-authored text: when authorship is unknown, participants exhibit a systematic preference for AI-generated content. These findings advance both theoretical understanding of human-AI text perception and practical design of linguistically grounded, cognitively aligned generative systems.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: Generation
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Prior studies have shown that distinguishing text generated by large language models (LLMs) from human-written one is highly challenging, and often no better than random guessing. To verify the generalizability of this finding across languages and domains, we perform an extensive case study to identify the upper bound of human detection accuracy. Across 16 datasets covering 9 languages and 9 domains, 19 annotators achieved an average detection accuracy of 87.6%, thus challenging previous conclusions. We find that major gaps between human and machine text lie in concreteness, cultural nuances, and diversity. Prompting by explicitly explaining the distinctions in the prompts can partially bridge the gaps in over 50% of the cases. However, we also find that humans do not always prefer human-written text, particularly when they cannot clearly identify its source.

Problem

• Human detection accuracy across languages
• Gaps between human and machine text
• Human preference for text source ambiguity

Innovation

• Multilingual human detection study
• Prompting bridges text gaps
• Human preference varies by source