Benchmark of stylistic variation in LLM-generated texts

2025-09-12

AI Summary

This study systematically investigates systematic stylistic divergences—specifically register variation—between LLM-generated and human-written texts. Method: Leveraging Biber’s Multidimensional Analysis (MDA), we construct two bilingual comparable corpora—AI-Brown (English) and AI-Koditex (Czech)—enabling the first explainable, multidimensional, cross-lingual benchmark for evaluating LLMs on register variation. We quantitatively assess 16 state-of-the-art base and instruction-tuned models across linguistic dimensions including information density, interactivity, and grammatical complexity. Contribution/Results: Instruction tuning significantly reduces stylistic divergence between model outputs and human writing along key register dimensions. Our work introduces the first register-variation-oriented, comparable LLM benchmark, supporting fine-grained model diagnosis and ranking. It establishes a novel paradigm for advancing research on LLM stylistic controllability and register adaptability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This study investigates the register variation in texts written by humans and comparable texts produced by large language models (LLMs). Biber's multidimensional analysis (MDA) is applied to a sample of human-written texts and AI-created texts generated to be their counterparts to find the dimensions of variation in which LLMs differ most significantly and most systematically from humans. As textual material, a new LLM-generated corpus AI-Brown is used, which is comparable to BE-21 (a Brown family corpus representing contemporary British English). Since all languages except English are underrepresented in the training data of frontier LLMs, similar analysis is replicated on Czech using AI-Koditex corpus and Czech multidimensional model. Examined were 16 frontier models in various settings and prompts, with emphasis placed on the difference between base models and instruction-tuned models. Based on this, a benchmark is created through which models can be compared with each other and ranked in interpretable dimensions.

Problem

• Compares stylistic variation in human versus LLM-generated texts
• Identifies key dimensions where LLMs differ systematically from humans
• Creates benchmark for model comparison across languages and settings

Innovation

• Using Biber's multidimensional analysis method
• Creating comparable AI-generated corpus AI-Brown
• Developing interpretable benchmark for model comparison