How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework

2026-05-22

AI Summary

Current evaluations of large language models predominantly emphasize factual accuracy and task-specific performance, often overlooking the linguistic human-likeness of their generated text. This work proposes the first register-aware linguistic evaluation framework, which systematically assesses the naturalness of model outputs by comparing the distributions of 67 Biber lexico-grammatical features between model-generated text and human-written corpora within specific registers, using Maximum Mean Discrepancy as the divergence metric. An evaluation across five English registers on seven open-source instruction-tuned models reveals that all models significantly deviate from human baselines, with the best-performing model varying by register. These findings indicate that register-specific factors account for differences in human-likeness more substantially than model scale alone.

Abstract

While factual correctness and task-performance have been in focus of Large Language Model (LLM) research for a long time, the fundamental question of how human-like generated texts are on a linguistic level has been underexplored. From a corpus-linguistic perspective, language production is inherently context-dependent, with distinct communicative contexts giving rise to differences in frequencies and co-occurrence patterns of linguistic features. A text failing to adhere to these patterns can be content-wise correct, but still be unfavorable to human readers. In this work, we propose a context-aware evaluation framework in which human-likeness is assessed using a two-sample problem between the linguistic feature distribution of a human reference corpus for a given register and a corresponding LLM-generated corpus. We implement this framework using the Maximum Mean Discrepancy (MMD) and the 67 lexico-grammatical features introduced by Biber, which are commonly applied in corpus linguistics. In our experiments, we compare seven instruction-tuned, open-source models across five English-language datasets spanning distinct registers against a human baseline. While across all tested setups, LLMs deviate from the human baseline, which models are closest to human language depends on the register and is not dictated by model size.

Problem

• human-likeness
• large language models
• register
• linguistic evaluation
• corpus linguistics

Innovation

• register-aware evaluation
• linguistic human-likeness
• Maximum Mean Discrepancy
• Biber dimensions
• corpus linguistics