SegMo: Segment-aligned Text to 3D Human Motion Generation

2025-12-24

AI Summary

Existing text-to-3D human motion generation methods enforce alignment only at the sequence level, neglecting intra-modal semantic structure. This work proposes a **segment-level alignment paradigm**, decomposing both text and motion into semantically coherent temporal segments and establishing cross-modal segment-to-segment correspondences. The method comprises three core components: text segmentation and feature extraction, motion segmentation and boundary detection, and segment-level contrastive alignment—unified within a shared embedding space to enable fine-grained cross-modal retrieval. Evaluated on HumanML3D, our approach achieves a TOP-1 retrieval accuracy of 0.553, substantially outperforming sequence-level baselines. Moreover, it supports motion localization and bidirectional text–motion retrieval. To our knowledge, this is the first work to empirically validate the effectiveness and generalizability of segment-level alignment for embodied generative tasks.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Multimodal Learning
• Computer Vision: Motion & Tracking

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Generating 3D human motions from textual descriptions is an important research problem with broad applications in video games, virtual reality, and augmented reality. Recent methods align the textual description with human motion at the sequence level, neglecting the internal semantic structure of modalities. However, both motion descriptions and motion sequences can be naturally decomposed into smaller and semantically coherent segments, which can serve as atomic alignment units to achieve finer-grained correspondence. Motivated by this, we propose SegMo, a novel Segment-aligned text-conditioned human Motion generation framework to achieve fine-grained text-motion alignment. Our framework consists of three modules: (1) Text Segment Extraction, which decomposes complex textual descriptions into temporally ordered phrases, each representing a simple atomic action; (2) Motion Segment Extraction, which partitions complete motion sequences into corresponding motion segments; and (3) Fine-grained Text-Motion Alignment, which aligns text and motion segments with contrastive learning. Extensive experiments demonstrate that SegMo improves the strong baseline on two widely used datasets, achieving an improved TOP 1 score of 0.553 on the HumanML3D test set. Moreover, thanks to the learned shared embedding space for text and motion segments, SegMo can also be applied to retrieval-style tasks such as motion grounding and motion-to-text retrieval.

Problem

• Generates 3D human motions from textual descriptions
• Aligns text and motion at a fine-grained segment level
• Improves alignment by decomposing descriptions and motions into segments

Innovation

• Decomposes text and motion into semantic segments
• Aligns text and motion segments with contrastive learning
• Learns shared embedding space for retrieval tasks