Multi-scale Coarse-to-fine Modeling for Test-time Human Motion Control

2026-05-14

AI Summary

Existing human motion generation methods suffer from limited controllability, slow inference speed, and inadequate adaptation to discrete sampling during testing. To address these limitations, this work proposes MSCoT, a novel model that adopts a multi-scale coarse-to-fine hierarchical modeling paradigm. MSCoT simultaneously predicts complete token sequences across multiple temporal scales and integrates an efficient multi-scale token guidance strategy with a lightweight differentiable refiner to enable test-time optimization. Evaluated on benchmarks such as HumanML3D, the proposed method achieves state-of-the-art performance, improving FID by 48%, reducing control error by 61%, and accelerating inference by 10× compared to diffusion-based approaches.

Abstract

We present MSCoT, a multi-scale, coarse-to-fine model for test-time human motion synthesis and control. Unlike recent approaches that rely on multiple iterative denoising/token-prediction steps, or modules tailored for specific control signals, MSCoT discretizes motion into a multi-scale hierarchical representation and predicts the entire token sequence at each temporal scale in a coarse-to-fine fashion. Building on this coarse-to-fine paradigm, we propose an efficient multi-scale token guidance strategy that overcomes the challenge of discrete sampling and steers the token distribution towards the control goals, allowing for fast and flexible control. To address the limitations of a discrete codebook, a lightweight token refiner further adds continuous residuals to the discrete token embeddings and allows differentiable test-time refinement optimization to ensure precise alignment with the control objectives. MSCoT is able to produce quality motions, consistent with the control constraints, while offering substantially faster sampling than diffusion-based approaches. Experiments on popular benchmarks demonstrate state-of-the-art controllable text-to-motion generation performance of MSCoT over existing baselines, with better motion quality (48% FID improvement), higher control accuracy (-61% avg error), and $10 \times$ faster inference speed on HumanML3D.

Problem

• human motion control
• test-time synthesis
• controllable generation
• motion quality
• inference speed

Innovation

• multi-scale modeling
• coarse-to-fine generation
• token guidance
• discrete-continuous refinement
• test-time optimization