QUDsim: Quantifying Discourse Similarities in LLM-Generated Text

2025-04-12

AI Summary

Large language models (LLMs) often implicitly reuse discourse structures, undermining textual creativity and diversity; existing similarity metrics—relying primarily on lexical or syntactic overlap—fail to capture deep discourse progression. This paper introduces QUDsim, the first computationally grounded discourse similarity metric formalizing the Questions Under Discussion (QUD) framework. QUDsim quantifies cross-document discourse structural similarity through discourse semantic modeling, automatic QUD tree inference, graph edit distance, and structural embedding. Crucially, it explicitly decouples structural similarity from content similarity. Empirical analysis reveals that LLMs reuse a narrower set of discourse patterns significantly more frequently than humans, and their dominant discourse structures markedly deviate from human distributions. QUDsim thus establishes a novel, theoretically grounded benchmark for evaluating generative text along dimensions of creativity, diversity, and human alignment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

As large language models become increasingly capable at various writing tasks, their weakness at generating unique and creative content becomes a major liability. Although LLMs have the ability to generate text covering diverse topics, there is an overall sense of repetitiveness across texts that we aim to formalize and quantify via a similarity metric. The familiarity between documents arises from the persistence of underlying discourse structures. However, existing similarity metrics dependent on lexical overlap and syntactic patterns largely capture $ extit{content}$ overlap, thus making them unsuitable for detecting $ extit{structural}$ similarities. We introduce an abstraction based on linguistic theories in Questions Under Discussion (QUD) and question semantics to help quantify differences in discourse progression. We then use this framework to build $ extbf{QUDsim}$, a similarity metric that can detect discursive parallels between documents. Using QUDsim, we find that LLMs often reuse discourse structures (more so than humans) across samples, even when content differs. Furthermore, LLMs are not only repetitive and structurally uniform, but are also divergent from human authors in the types of structures they use.

Problem

• Quantifying discourse similarities in LLM-generated text
• Detecting structural similarities beyond lexical overlap
• Comparing discourse structures between LLMs and humans

Innovation

• QUD-based abstraction for discourse analysis
• QUDsim metric detects structural similarities
• Compares LLM and human discourse structures