DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control

2024-10-07

AI Summary

Existing text-driven human motion generation methods suffer from limitations in sequence length, offline processing, and absence of spatial constraints, hindering long-horizon synthesis, real-time responsiveness, and scene interaction. This paper introduces the first unified framework integrating latent diffusion models with autoregressive motion primitives, enabling concurrent natural language streaming input and 3D geometric conditioning—including target positions and scene layouts. By performing noise optimization in the latent space, the method supports online motion generation; combined with MDP-based reinforcement learning, it achieves spatially controllable motion planning. The approach guarantees millisecond-level latency while significantly improving long-range semantic coherence and motion realism. Quantitative and qualitative evaluations demonstrate state-of-the-art performance across generation quality, computational efficiency, and fine-grained spatial controllability.

Technology Category

• Natural Language Processing: Generation
• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Text-conditioned human motion generation, which allows for user interaction through natural language, has become increasingly popular. Existing methods typically generate short, isolated motions based on a single input sentence. However, human motions are continuous and can extend over long periods, carrying rich semantics. Creating long, complex motions that precisely respond to streams of text descriptions, particularly in an online and real-time setting, remains a significant challenge. Furthermore, incorporating spatial constraints into text-conditioned motion generation presents additional challenges, as it requires aligning the motion semantics specified by text descriptions with geometric information, such as goal locations and 3D scene geometry. To address these limitations, we propose DartControl, in short DART, a Diffusion-based Autoregressive motion primitive model for Real-time Text-driven motion control. Our model effectively learns a compact motion primitive space jointly conditioned on motion history and text inputs using latent diffusion models. By autoregressively generating motion primitives based on the preceding history and current text input, DART enables real-time, sequential motion generation driven by natural language descriptions. Additionally, the learned motion primitive space allows for precise spatial motion control, which we formulate either as a latent noise optimization problem or as a Markov decision process addressed through reinforcement learning. We present effective algorithms for both approaches, demonstrating our model's versatility and superior performance in various motion synthesis tasks. Experiments show our method outperforms existing baselines in motion realism, efficiency, and controllability. Video results are available on the project page: https://zkf1997.github.io/DART/.

Problem

• Generates long, complex motions from continuous text descriptions.
• Incorporates spatial constraints for precise motion control.
• Enables real-time, sequential motion generation using natural language.

Innovation

• Diffusion-based autoregressive motion model
• Real-time text-driven motion control
• Latent diffusion for motion primitive space