GRACE: Boosting Video MLLMs with Grounded Action-Centric Evidence for Viewer Sentiment Prediction

2026-06-15

AI Summary

This work addresses the challenge of predicting viewer sentiment in video advertisements, where fine-grained emotion-relevant behaviors and visual cues are difficult to capture from full-frame inputs. The authors propose an action-centric, structured evidence enhancement framework that extracts temporal subject-predicate-object triplets, crops visual patches of participating entities, and integrates visible text to construct explicit, spatially localizable multimodal reasoning cues. This approach uniquely combines action triplets with entity-specific visual crops to guide interpretable sentiment reasoning in multimodal large language models (Qwen2.5-VL/Qwen3-VL). Evaluated on the Pitts dataset, the method significantly outperforms baseline approaches, and transfer experiments on AdsQA and TVQA subsets demonstrate its strong generalization capability.

Abstract

Viewer sentiment prediction in video advertisements aims to infer the latent affective response evoked in the audience. To bridge the gap between what is shown and what is felt, models must deduce hidden viewer emotions from explicit visual narratives, concrete character-object interactions, and visible textual cues. However, standard Multimodal Large Language Models (MLLMs) typically rely on holistic frame representations, which leave these fine-grained, affect-relevant events implicit and complicate precise emotional reasoning. To address this, we propose a grounded action-centric evidence augmentation framework that enhances video MLLMs' clue extraction and comprehension by introducing explicit event structure and localized visual evidence. Our method extracts temporally ordered subject-verb-object (SVO) triplets and auxiliary visible textual cues from action-centric video descriptions, grounds subject and object entities as visual entity crops, and then enables the MLLM to perform clue-enhanced emotional reasoning based on these extracted structured clues. In this way, action triplets specify "what happens", while grounded visual entity crops anchor "who or what participates in each event" to concrete visual evidence. Experiments on the Pitts dataset show consistent improvements over Qwen2.5-VL and Qwen3-VL baselines. Ablation studies, cross-dataset evaluation on AdsQA, and transfer experiments on an emotion-focused TVQA subset further support the effectiveness and generalization of our approach.

Problem

• viewer sentiment prediction
• video advertisements
• multimodal large language models
• fine-grained event understanding
• emotional reasoning

Innovation

• grounded visual evidence
• action-centric reasoning
• structured SVO triplets
• video MLLMs
• viewer sentiment prediction