Video2Reaction: Mapping Video to Audience Reaction Distribution in the Wild

2026-07-07

AI Summary

This study addresses the challenge of predicting the distribution of viewer emotional responses elicited by videos in real-world settings, with applications in content creation, recommendation, and media analysis. To this end, we introduce Video2Reaction, a large-scale multimodal dataset comprising over 10,000 short videos paired with their corresponding social media emotion distributions. We propose a two-stage, multi-agent annotation pipeline based on open-source large language models, achieving efficient and low-cost labeling with 86% accuracy validated by human annotators. We establish the first benchmark for video-to-emotion-distribution prediction, revealing the limited performance of existing video foundation models under zero-shot settings. Fine-tuning LLaVA-Next yields a Top-3 F1 score of 77% for dominant emotion prediction, substantially outperforming zero-shot approaches.

Abstract

Understanding and forecasting audience reactions to video content are crucial for improving content creation, recommendation systems, and media analysis. To enable audience reaction prediction and other content engagement applications, we introduce $\textbf{Video2Reaction}$, a multimodal dataset that maps short movie segments to a distribution of $\textit{induced emotions}$ of viewers in the wild, as expressed through social media. $\textbf{Video2Reaction}$ spans more than 10,000 videos and serves as a reliable benchmark as well as a training resource for audience reaction prediction. To enable cost-effective continuous annotations as reactions may change over time, we develop a two-stage multi-agent pipeline using only open-source LLMs, achieving 86% correctness under blind human verification despite the inherently noisy and subjective nature of the task. We establish the first benchmark for video-to-reaction-distribution prediction in the wild and show that pretrained foundation video models fail in zero-shot settings, while finetuning transforms them into state-of-the-art predictors capable of modeling both full reaction distributions and dominant responses from video alone. However, the task remains challenging: even the strongest methods achieve only 77% Top-3 F1 in dominant reaction prediction (LLaVA-Next), highlighting a substantial gap in modeling collective audience reaction. \modification{Dataset and code are available at our project page: https://information-fusion-lab-umass.github.io/video2reaction-bench.github.io

Problem

• audience reaction
• emotion prediction
• video understanding
• reaction distribution
• multimodal learning

Innovation

• audience reaction prediction
• multimodal dataset
• emotion distribution
• multi-agent annotation
• video foundation models