MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities

2025-04-03

AI Summary

Existing motion-language models are constrained to coarse-grained motion-text modeling, limiting their capability for limb-level fine-grained understanding and generation. To address this, we propose MGM—the first unified multi-granularity motion-language model—built upon a large language model architecture and integrating motion representation learning with multi-task joint optimization. MGM introduces a novel multi-granularity collaborative training paradigm, jointly optimizing auxiliary tasks including temporal localization, fine-grained action description, hierarchical annotation, and cross-granularity bidirectional alignment. It achieves state-of-the-art performance on both text-to-motion and motion-to-text generation. Crucially, we systematically demonstrate, for the first time, MGM’s effectiveness on emerging fine-grained action understanding, local motion editing, and controllable generation tasks—significantly advancing beyond conventional single-granularity modeling paradigms.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Multiagent Systems: Multiagent Learning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent motion-aware large language models have demonstrated promising potential in unifying motion comprehension and generation. However, existing approaches primarily focus on coarse-grained motion-text modeling, where text describes the overall semantics of an entire motion sequence in just a few words. This limits their ability to handle fine-grained motion-relevant tasks, such as understanding and controlling the movements of specific body parts. To overcome this limitation, we pioneer MG-MotionLLM, a unified motion-language model for multi-granular motion comprehension and generation. We further introduce a comprehensive multi-granularity training scheme by incorporating a set of novel auxiliary tasks, such as localizing temporal boundaries of motion segments via detailed text as well as motion detailed captioning, to facilitate mutual reinforcement for motion-text modeling across various levels of granularity. Extensive experiments show that our MG-MotionLLM achieves superior performance on classical text-to-motion and motion-to-text tasks, and exhibits potential in novel fine-grained motion comprehension and editing tasks. Project page: CVI-SZU/MG-MotionLLM

Problem

• Unifies motion comprehension and generation across multiple granularities
• Addresses limitations in fine-grained motion-text modeling
• Enhances motion understanding and control for specific body parts

Innovation

• Unified motion-language model for multi-granular tasks
• Multi-granularity training with novel auxiliary tasks
• Superior performance in fine-grained motion tasks