Emotion Understanding in Streaming Video with Trajectory-Aware Reliability

2026-08-27

AI Summary

研究实时视频情绪理解问题，提出TRACE框架，通过音频前缀形成情绪信念，并根据置信度、熵、稳定性和类别切换模式估计可靠性，以优化准确性和成本之间的权衡。

Abstract

Video emotion understanding is commonly studied as an offline classification problem, where the complete video segment is available before prediction. Real-time interaction, however, requires emotion decisions from incomplete and evolving evidence. This paper studies streaming video emotion understanding as a reliability-aware decision process over evolving emotion beliefs. In this setting, a single confident prefix prediction can still be unreliable when the underlying belief trajectory is unstable or repeatedly switches across emotion classes. We propose TRACE, a trajectory-aware reliability framework that forms low-latency emotion beliefs from streaming audio prefixes, estimates reliability from confidence, entropy, stability, and class-switching patterns, and selectively invokes contextual belief reinterpretation with visual, textual, and neighboring-utterance evidence. TRACE keeps stable cases in the low-latency online pathway while allocating stronger multimodal reasoning to uncertain cases that remain ambiguous. Experiments on StreamMER, MELD, and MER2024 show that TRACE improves the accuracy-cost trade-off, retaining most full-context gains while reducing unnecessary contextual reasoning.

Problem

• Streaming Video
• Emotion Understanding
• Real-time Interaction
• Evolving Evidence
• Reliability

Innovation

• trajectory-aware reliability
• streaming video emotion understanding
• multimodal reasoning
• confidence and entropy analysis
• contextual belief reinterpretation