Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward

2025-05-31

AI Summary

This work addresses text-driven human motion generation, aiming to enhance precise natural language control over complex, anthropomorphic action sequences. We propose an LLM-empowered semantic alignment paradigm and introduce an end-to-end text-to-motion framework that jointly optimizes text embeddings and joint trajectories via a hybrid architecture integrating autoregressive modeling, diffusion processes, and Transformer-based sequence learning. We establish, for the first time, a comprehensive three-dimensional evaluation framework assessing generation quality, efficiency, and controllability, and systematically chart the technical evolution of the field. Experiments demonstrate significant improvements in motion semantic fidelity and contextual coherence. The approach exhibits strong practical potential in medical rehabilitation, humanoid robotics, and animation production, offering a novel methodological foundation for lightweight and efficient motion generation.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Motion & Tracking

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

This paper presents an in-depth survey on the use of multimodal Generative Artificial Intelligence (GenAI) and autoregressive Large Language Models (LLMs) for human motion understanding and generation, offering insights into emerging methods, architectures, and their potential to advance realistic and versatile motion synthesis. Focusing exclusively on text and motion modalities, this research investigates how textual descriptions can guide the generation of complex, human-like motion sequences. The paper explores various generative approaches, including autoregressive models, diffusion models, Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and transformer-based models, by analyzing their strengths and limitations in terms of motion quality, computational efficiency, and adaptability. It highlights recent advances in text-conditioned motion generation, where textual inputs are used to control and refine motion outputs with greater precision. The integration of LLMs further enhances these models by enabling semantic alignment between instructions and motion, improving coherence and contextual relevance. This systematic survey underscores the transformative potential of text-to-motion GenAI and LLM architectures in applications such as healthcare, humanoids, gaming, animation, and assistive technologies, while addressing ongoing challenges in generating efficient and realistic human motion.

Problem

• Exploring multimodal GenAI for human motion understanding and generation
• Investigating text-guided synthesis of complex human-like motion sequences
• Assessing generative models' performance in text-to-motion applications

Innovation

• Multimodal GenAI for human motion synthesis
• Autoregressive LLMs enhance semantic motion alignment
• Text-conditioned models improve motion precision