Choreographing a World of Dynamic Objects

2026-01-07

arXiv.org

AI Summary

Existing methods for 4D dynamic scene generation either rely on class-specific rules, limiting their scalability, or require large-scale 4D datasets, which constrains their generalization. This work proposes CHORD, a universal generative framework that, for the first time, brings the versatility of video generation models to 4D dynamic object synthesis without requiring category priors or extensive 4D training data. By distilling Eulerian representations from 2D videos, CHORD extracts Lagrangian motion information to enable category-agnostic modeling of multi-object 4D dynamics. The framework supports complex multi-body interactions and robotic policy generation, significantly outperforming existing approaches in both synthesis quality and generalization capability.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Dynamic objects in our physical 4D (3D + time) world are constantly evolving, deforming, and interacting with other objects, leading to diverse 4D scene dynamics. In this paper, we present a universal generative pipeline, CHORD, for CHOReographing Dynamic objects and scenes and synthesizing this type of phenomena. Traditional rule-based graphics pipelines to create these dynamics are based on category-specific heuristics, yet are labor-intensive and not scalable. Recent learning-based methods typically demand large-scale datasets, which may not cover all object categories in interest. Our approach instead inherits the universality from the video generative models by proposing a distillation-based pipeline to extract the rich Lagrangian motion information hidden in the Eulerian representations of 2D videos. Our method is universal, versatile, and category-agnostic. We demonstrate its effectiveness by conducting experiments to generate a diverse range of multi-body 4D dynamics, show its advantage compared to existing methods, and demonstrate its applicability in generating robotics manipulation policies. Project page: https://yanzhelyu.github.io/chord

Problem

• 4D dynamics
• dynamic objects
• scene generation
• category-agnostic modeling
• motion synthesis

Innovation

• 4D dynamics
• generative modeling
• motion distillation
• category-agnostic synthesis
• Lagrangian motion extraction