Marrying Text-to-Motion Generation with Skeleton-Based Action Recognition

2026-04-18

AI Summary

Existing approaches typically treat action recognition and text-driven motion generation as separate tasks, overlooking their intrinsic semantic connections. This work proposes CoAMD—a skeleton-coordinate-based autoregressive motion diffusion model—that unifies both tasks within a single framework for the first time. CoAMD leverages a multimodal action recognizer to provide semantic gradient guidance and synthesizes motions through a coarse-to-fine strategy. Furthermore, it establishes a unified training and evaluation framework across tasks using absolute coordinates. Evaluated on 13 benchmarks spanning action recognition, text-to-motion generation, text-motion retrieval, and motion editing, the method achieves state-of-the-art performance across all four tasks, demonstrating its effectiveness and generalizability.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Multimodal Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Human action recognition and motion generation are two active research problems in human-centric computer vision, both aiming to align motion with textual semantics. However, most existing works study these two problems separately, without uncovering the links between them, namely that motion generation requires semantic comprehension. This work investigates unified action recognition and motion generation by leveraging skeleton coordinates for both motion understanding and generation. We propose Coordinates-based Autoregressive Motion Diffusion (CoAMD), which synthesizes motion in a coarse-to-fine manner. As a core component of CoAMD, we design a Multi-modal Action Recognizer (MAR) that provides gradient-based semantic guidance for motion generation. Furthermore, we establish a rigorous benchmark by evaluating baselines on absolute coordinates. Our model can be applied to four important tasks, including skeleton-based action recognition, text-to-motion generation, text-motion retrieval, and motion editing. Extensive experiments on 13 benchmarks across these tasks demonstrate that our approach achieves state-of-the-art performance, highlighting its effectiveness and versatility for human motion modeling. Code is available at https://github.com/jidongkuang/CoAMD.

Problem

• text-to-motion generation
• skeleton-based action recognition
• semantic comprehension
• human motion modeling

Innovation

• text-to-motion generation
• skeleton-based action recognition
• motion diffusion
• multimodal semantic guidance
• coarse-to-fine synthesis