ByteLoom: Weaving Geometry-Consistent Human-Object Interactions through Progressive Curriculum Learning

2025-12-28

AI Summary

Existing HOI video generation methods suffer from two key limitations: (1) cross-view geometric inconsistency of objects and (2) excessive reliance on fine-grained hand mesh annotations. To address these, we propose a hand-mesh-free multi-view HOI video generation framework. First, we introduce an RCM-cache mechanism to jointly encode object geometry and precisely model 6-DoF spatial transformations. Second, we design a Diffusion Transformer (DiT)-based generative architecture that fuses Relative Coordinate Maps (RCMs) with 3D object encodings. Third, we adopt a progressive curriculum learning strategy to mitigate data scarcity and reduce annotation dependency. Experiments demonstrate significant improvements in cross-view geometric consistency, interaction plausibility, and identity stability—while preserving motion smoothness and fine-grained manipulation capability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Human-object interaction (HOI) video generation has garnered increasing attention due to its promising applications in digital humans, e-commerce, advertising, and robotics imitation learning. However, existing methods face two critical limitations: (1) a lack of effective mechanisms to inject multi-view information of the object into the model, leading to poor cross-view consistency, and (2) heavy reliance on fine-grained hand mesh annotations for modeling interaction occlusions. To address these challenges, we introduce ByteLoom, a Diffusion Transformer (DiT)-based framework that generates realistic HOI videos with geometrically consistent object illustration, using simplified human conditioning and 3D object inputs. We first propose an RCM-cache mechanism that leverages Relative Coordinate Maps (RCM) as a universal representation to maintain object's geometry consistency and precisely control 6-DoF object transformations in the meantime. To compensate HOI dataset scarcity and leverage existing datasets, we further design a training curriculum that enhances model capabilities in a progressive style and relaxes the demand of hand mesh. Extensive experiments demonstrate that our method faithfully preserves human identity and the object's multi-view geometry, while maintaining smooth motion and object manipulation.

Problem

• Generates geometry-consistent human-object interaction videos
• Reduces reliance on fine-grained hand mesh annotations
• Enhances cross-view consistency with multi-view object information

Innovation

• RCM-cache mechanism for geometry consistency
• Progressive curriculum learning for dataset scarcity
• Simplified human conditioning with 3D object inputs