Rhetorical Questions in LLM Representations: A Linear Probing Study

2026-04-15

AI Summary

This study investigates how large language models internally represent rhetorical questions and distinguish them from information-seeking questions. By applying linear probes to model representations across layers on two social media datasets, the authors analyze the encoding mechanisms of rhetorical questions. The findings reveal that rhetorical questions are not encoded along a single shared direction but instead rely on multiple linear directions that separately capture discourse stance, syntactic structure, and other cues. Rhetorical questions are most stably separable in the final-layer representations of the last token, achieving cross-dataset detection AUROC scores between 0.7 and 0.8. However, probe directions trained on different datasets exhibit less than 0.2 overlap, highlighting the high diversity and context dependence of these internal representations.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Rhetorical questions are asked not to seek information but to persuade or signal stance. How large language models internally represent them remains unclear. We analyze rhetorical questions in LLM representations using linear probes on two social-media datasets with different discourse contexts, and find that rhetorical signals emerge early and are most stably captured by last-token representations. Rhetorical questions are linearly separable from information-seeking questions within datasets, and remain detectable under cross-dataset transfer, reaching AUROC around 0.7-0.8. However, we demonstrate that transferability does not simply imply a shared representation. Probes trained on different datasets produce different rankings when applied to the same target corpus, with overlap among the top-ranked instances often below 0.2. Qualitative analysis shows that these divergences correspond to distinct rhetorical phenomena: some probes capture discourse-level rhetorical stance embedded in extended argumentation, while others emphasize localized, syntax-driven interrogative acts. Together, these findings suggest that rhetorical questions in LLM representations are encoded by multiple linear directions emphasizing different cues, rather than a single shared direction.

Problem

• rhetorical questions
• large language models
• representation
• linear probing
• transferability

Innovation

• rhetorical questions
• linear probing
• representation analysis
• cross-dataset transfer
• large language models