DIMO: Diverse 3D Motion Generation for Arbitrary Objects

2025-11-10

AI Summary

This work addresses the challenge of generating diverse 3D object motions from a single input image. To leverage generalizable motion priors, the method distills knowledge from a pretrained video diffusion model and maps it into a shared, low-dimensional latent space. Motion structure is then explicitly modeled via neural keypoint trajectories, which drive 3D Gaussian splatting to jointly represent geometry and appearance. The core contribution is a structured, compact motion latent representation that enables diverse motion sequence generation in a single forward pass, while supporting interactive capabilities such as motion interpolation and language-guided motion synthesis. Experiments demonstrate that the approach achieves high-fidelity 3D reconstruction and temporal coherence, alongside significantly improved inference efficiency. Overall, it establishes a novel paradigm for controllable, single-image-driven 3D content generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We present DIMO, a generative approach capable of generating diverse 3D motions for arbitrary objects from a single image. The core idea of our work is to leverage the rich priors in well-trained video models to extract the common motion patterns and then embed them into a shared low-dimensional latent space. Specifically, we first generate multiple videos of the same object with diverse motions. We then embed each motion into a latent vector and train a shared motion decoder to learn the distribution of motions represented by a structured and compact motion representation, i.e., neural key point trajectories. The canonical 3D Gaussians are then driven by these key points and fused to model the geometry and appearance. During inference time with learned latent space, we can instantly sample diverse 3D motions in a single-forward pass and support several interesting applications including 3D motion interpolation and language-guided motion generation. Our project page is available at https://linzhanm.github.io/dimo.

Problem

• Generate diverse 3D motions for arbitrary objects from single images
• Create structured motion representation using neural key point trajectories
• Enable 3D motion interpolation and language-guided motion generation

Innovation

• Leverages video models for motion pattern extraction
• Embeds motions into shared latent space representation
• Uses neural key points to drive 3D Gaussians