Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialogue

2026-09-14

AI Summary

研究通过分析27组虚拟现实对话中的多模态行为，探讨了否定线索的非语言表现形式及其时间分布，使用时间序列模型分类方法，发现面部特征是主要信息载体。

Abstract

Negation is typically modeled through its linguistic realization, although spoken interaction is accompanied by tightly coordinated nonverbal behavior. We ask whether contexts centered on spoken negation cues contain measurable multimodal behavioral information: whether they can be distinguished from matched control contexts without lexical or acoustic input, where this information occurs in time, which modalities carry it, and whether it extends to the dialogue partner. We study 27 human-human interviews conducted in virtual reality, comprising temporally aligned gaze, facial, head, body, hand, and finger behavior and 964 annotated negation cues. Treating classification as a predictive probe, we compare 20 time-series models while excluding lexical and acoustic information, and then systematically vary temporal context, interactional source, modality availability, and event timing. Across grouped 10-fold cross-validation, the strongest probes reach up to .75 mean held-out AUROC from speaker-side behavior. Temporal analyses show that predictive information is concentrated around cue onset but remains detectable over a broader surrounding interval, while dialogue-partner behavior carries weaker predictive information with a comparatively diffuse temporal profile. Ablation and timing perturbations further show that facial features produce the largest modality-ablation effect and that the trained probe is sensitive to the temporal organization of the observed events.

Problem

• Negation
• Multimodal Behavior
• Dialogue
• Nonverbal Cues
• Temporal Analysis

Innovation

• multimodal behavior
• negation cues
• temporal analysis
• nonverbal communication
• facial features