SparseCtrl-HOI: Sparse Temporal Control for Human-Object Interaction Video Generation

2026-07-07

AI Summary

Existing methods for human-object interaction (HOI) video generation rely on densely annotated temporal labels, which are costly to acquire and limit action diversity. This work proposes the first HOI video generation framework based on sparse keyframes, capable of synthesizing high-quality, physically plausible interaction videos from only a few annotated frames. The approach introduces a temporal anchoring mechanism and a Time-Controlled Rotary Positional Embedding (TiRoPE) to ensure spatiotemporal consistency, while leveraging a multimodal large language model (MLLM) to inject high-level motion priors for natural and smooth action transitions. Experiments demonstrate that the method substantially reduces annotation costs and outperforms existing approaches in generation quality, showing strong applicability in scenarios such as e-commerce live streaming. The project also releases SparseHOI-5K, a high-quality dataset, along with open-source code.

Abstract

Human-Object Interaction (HOI) video generation aims to synthesize realistic videos of humans manipulating diverse objects, serving as a promising avenue for AI-driven live streaming e-commerce. A primary obstacle in this domain lies in the complexity of modeling fine-grained physical dynamics and the intricate spatial-temporal coordination between human hands and objects. Existing approaches to this problem typically rely on dense temporal guidance, e.g., frame-wise hand-object pose sequences, to strictly control the interaction process. However, such dense guidance incurs high annotation costs and affects motion synthesis diversity. To overcome these limitations, we introduce SparseCtrl-HOI, a novel sparse temporal control framework for HOI video generation. It requires only a few keyframes that capture interaction states at designated timestamps. Specifically, we employ a Time-Controlled Rotary Positional Embedding (TiRoPE) mechanism to temporally anchor these keyframes while preserving their spatial integrity. Subsequently, to govern the dynamics across intermediate frames, we propose a Motion Prior Injection Module that leverages Multimodal Large Language Models (MLLMs) to extract high-level motion priors. This empowers the model to hallucinate logically and physically plausible transitions. Furthermore, we build SparseHOI-5K, a high-quality and richly annotated dataset for HOI video generation with sparse temporal control. Comprehensive evaluations confirm that our method substantially reduces annotation overhead while synthesizing superior live-streaming e-commerce videos. Both our code and dataset are publicly available at https://mpi-lab.github.io/SparseCtrl-HOI.

Problem

• Human-Object Interaction
• Video Generation
• Temporal Control
• Sparse Guidance
• Motion Synthesis

Innovation

• Sparse Temporal Control
• Human-Object Interaction
• Video Generation
• Motion Prior Injection
• Multimodal Large Language Models