EgoReAct: Egocentric Video-Driven 3D Human Reaction Generation

2025-12-28

AI Summary

This work addresses the lack of temporal causality and spatial misalignment in 3D human reactive motion generation from first-person videos. To this end, we propose the first autoregressive, causally constrained framework for 3D reactive motion synthesis. Methodologically: (1) we introduce the Human Reaction Dataset (HRD), the first spatially aligned benchmark for reactive motion; (2) we enhance 3D spatial grounding via joint depth estimation and head dynamics modeling; and (3) we integrate VQ-VAE with a GPT-based architecture to enable real-time, causal sequence modeling of 3D motion dynamics. Experiments demonstrate that our approach significantly outperforms state-of-the-art methods in motion plausibility, spatial consistency, and inference efficiency, while strictly enforcing temporal causality. All code, pretrained models, and the HRD dataset will be publicly released.

Technology Category

• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Humans exhibit adaptive, context-sensitive responses to egocentric visual input. However, faithfully modeling such reactions from egocentric video remains challenging due to the dual requirements of strictly causal generation and precise 3D spatial alignment. To tackle this problem, we first construct the Human Reaction Dataset (HRD) to address data scarcity and misalignment by building a spatially aligned egocentric video-reaction dataset, as existing datasets (e.g., ViMo) suffer from significant spatial inconsistency between the egocentric video and reaction motion, e.g., dynamically moving motions are always paired with fixed-camera videos. Leveraging HRD, we present EgoReAct, the first autoregressive framework that generates 3D-aligned human reaction motions from egocentric video streams in real-time. We first compress the reaction motion into a compact yet expressive latent space via a Vector Quantised-Variational AutoEncoder and then train a Generative Pre-trained Transformer for reaction generation from the visual input. EgoReAct incorporates 3D dynamic features, i.e., metric depth, and head dynamics during the generation, which effectively enhance spatial grounding. Extensive experiments demonstrate that EgoReAct achieves remarkably higher realism, spatial consistency, and generation efficiency compared with prior methods, while maintaining strict causality during generation. We will release code, models, and data upon acceptance.

Problem

• Generates 3D human reactions from egocentric video streams
• Addresses spatial misalignment between video and reaction motion
• Ensures real-time, causal generation with enhanced spatial grounding

Innovation

• Autoregressive framework for real-time 3D reaction generation
• Vector Quantised-VAE compresses motion into latent space
• Generative Pre-trained Transformer uses 3D dynamic features