Semantic Delta: An Interpretable Signal Differentiating Human and LLMs Dialogue

2026-03-20

AI Summary

This work proposes “Semantic Delta”—a lightweight, interpretable statistical metric for distinguishing between human- and large language model (LLM)-generated dialogue texts. The method leverages the Empath lexicon to map semantic category intensities and quantifies topical concentration by computing the difference in intensity between the two most prominent semantic categories within a dialogue. It introduces, for the first time, the concentration of semantic distributions as a zero-shot, low-cost discriminative signal, revealing that LLM-generated texts exhibit an overly rigid thematic structure compared to human discourse. Experimental results demonstrate that Semantic Delta values are significantly higher in LLM-generated texts than in human-written ones, and that incorporating this metric effectively enhances the performance of ensemble detection systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Do LLMs talk like us? This question intrigues a multitude of scholar and it is relevant in many fields, from education to academia. This work presents an interpretable statistical feature for distinguishing human written and LLMs generated dialogue. We introduce a lightweight metric derived from semantic categories distribution. Using the Empath lexical analysis framework, each text is mapped to a set of thematic intensity scores. We define semantic delta as the difference between the two most dominant category intensities within a dialogue, hypothesizing that LLM outputs exhibit stronger thematic concentration than human discourse. To evaluate this hypothesis, conversational data were generated from multiple LLM configurations and compared against heterogeneous human corpora, including scripted dialogue, literary works, and online discussions. A Welch t-test was applied to the resulting distributions of semantic delta values. Results show that AI-generated texts consistently produce higher deltas than human texts, indicating a more rigid topics structure, whereas human dialogue displays a broader and more balanced semantic spread. Rather than replacing existing detection techniques, the proposed zero-shot metric provides a computationally inexpensive complementary signal that can be integrated into ensemble detection systems. These finding also contribute to the broader empirical understanding of LLM behavioural mimicry and suggest that thematic distribution constitutes a quantifiable dimension along which current models fall short of human conversational dynamics.

Problem

• semantic delta
• human dialogue
• LLM-generated text
• thematic distribution
• dialogue analysis

Innovation

• Semantic Delta
• LLM detection
• interpretable metric
• thematic concentration
• zero-shot