DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering

2026-04-16

AI Summary

This study addresses the lack of systematic representation and comparison of rhetorical strategies employed by humans and large language models (LLMs) in retrieval-based question answering. It proposes DiscoTrace, a novel method grounded in Rhetorical Structure Theory that models answers as sequences of rhetorical acts paired with question interpretations, thereby establishing the first structured framework enabling direct comparison of response strategies across human communities and LLMs. Integrating discourse act annotation, question interpretation modeling, and cross-group analysis, the approach reveals significant variation in rhetorical strategies among nine distinct human communities. In contrast, even under imitation prompting, LLMs exhibit limited rhetorical diversity and consistently favor broad coverage over selective engagement with specific question interpretations.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We introduce DiscoTrace, a method to identify the rhetorical strategies that answerers use when responding to information-seeking questions. DiscoTrace represents answers as a sequence of question-related discourse acts paired with interpretations of the original question, annotated on top of rhetorical structure theory parses. Applying DiscoTrace to answers from nine different human communities reveals that communities have diverse preferences for answer construction. In contrast, LLMs do not exhibit rhetorical diversity in their answers, even when prompted to mimic specific human community answering guidelines. LLMs also systematically opt for breadth, addressing interpretations of questions that human answerers choose not to address. Our findings can guide the development of pragmatic LLM answerers that consider a range of strategies informed by context in QA.

Problem

• rhetorical strategies
• information-seeking question answering
• discourse acts
• LLM answer diversity
• human vs. LLM

Innovation

• DiscoTrace
• rhetorical strategy
• discourse act
• information-seeking QA
• LLM pragmatics