Fine-grained Human Motion Understanding with Language Models

2026-06-18

AI Summary

This work proposes a novel large language model–based approach for fine-grained human action understanding by modeling actions as skeletal pose sequences with explicit timestamps. The method employs a unified pose encoder that accommodates both 2D and 3D inputs without requiring ground-truth 3D motion capture data, and integrates video context to enhance semantic comprehension. Training leverages multi-task mixed supervision—including pose description and question answering—augmented with timestamp token embeddings to explicitly capture the temporal structure, duration, and rhythm of actions. The approach achieves state-of-the-art performance across multiple benchmarks, including BABEL-QA, HuMMan-QA, CompMo, NTU-RGB+D, and QEVD-Coach, surpassing prior methods that rely on 3D information despite using only 2D pose inputs.

Abstract

In this work, we propose \methodname, an LLM-based model for fine-grained human motion understanding that represents motion as a sequence of skeletal poses with explicit timestamps for each pose. Each pose encodes body joint positions and is temporally grounded with timestamp tokens, allowing the model to reason about motion order, duration, and rhythm. To study what supervision is needed for motion-language reasoning, we construct a diverse training mixture spanning pose captioning, pose question answering, motion captioning, and motion question answering. Our ablations show that the primary gains come from the diversity of pose- and motion-level supervision, while staged training provides a smaller additional benefit. Different from previous works that rely on ground-truth 3D motion capture, our approach supports both 2D and 3D skeletal motion representations through a unified pose encoder, and can optionally incorporate video to provide contextual information. Extensive experiments on BABEL-QA, HuMMan-QA, CompMo, NTU-RGB+D, and QEVD-Coach demonstrate that our method achieves state-of-the-art performance across multiple benchmarks, highlighting the effectiveness of explicit temporal encoding and diverse pose- and motion-level supervision for fine-grained human motion understanding. Notably, even when using only 2D skeletal input, our approach surpasses previous 3D-based methods.

Problem

• fine-grained human motion understanding
• temporal reasoning
• pose-language alignment
• skeletal motion representation
• motion-language reasoning

Innovation

• fine-grained motion understanding
• temporal grounding
• pose-language reasoning
• unified 2D/3D skeletal representation
• diverse supervision