Bridging Semantic and Kinematic Conditions with Diffusion-based Discrete Motion Tokenizer

2026-03-19

AI Summary

This work addresses the challenge of simultaneously achieving semantic controllability and kinematic fidelity in human motion generation. The authors propose a three-stage framework—perception, planning, and control—featuring a novel discrete motion tokenizer, MoTok, which, for the first time, integrates a diffusion mechanism into the tokenization process to enable decoupled modeling of semantic and kinematic conditions. This approach preserves high-level semantic abstraction while supporting fine-grained motion reconstruction. Experiments demonstrate state-of-the-art performance on HumanML3D, achieving a trajectory error as low as 0.08 cm and a Fréchet Inception Distance (FID) of 0.029; under stringent kinematic constraints, FID further improves to 0.014. Notably, the method surpasses MaskControl using only one-sixth the number of tokens, substantially enhancing both controllability and generation quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Prior motion generation largely follows two paradigms: continuous diffusion models that excel at kinematic control, and discrete token-based generators that are effective for semantic conditioning. To combine their strengths, we propose a three-stage framework comprising condition feature extraction (Perception), discrete token generation (Planning), and diffusion-based motion synthesis (Control). Central to this framework is MoTok, a diffusion-based discrete motion tokenizer that decouples semantic abstraction from fine-grained reconstruction by delegating motion recovery to a diffusion decoder, enabling compact single-layer tokens while preserving motion fidelity. For kinematic conditions, coarse constraints guide token generation during planning, while fine-grained constraints are enforced during control through diffusion-based optimization. This design prevents kinematic details from disrupting semantic token planning. On HumanML3D, our method significantly improves controllability and fidelity over MaskControl while using only one-sixth of the tokens, reducing trajectory error from 0.72 cm to 0.08 cm and FID from 0.083 to 0.029. Unlike prior methods that degrade under stronger kinematic constraints, ours improves fidelity, reducing FID from 0.033 to 0.014.

Problem

• motion generation
• semantic conditioning
• kinematic control
• discrete tokenization
• diffusion models

Innovation

• diffusion-based tokenizer
• discrete motion representation
• semantic-kinematic decoupling
• motion generation
• token efficiency