A Plug-and-Play 2D Motion Interface for Real-World Motion Language Models

2026-08-16

AI Summary

This study addresses the challenge that acquiring 3D motion from monocular videos hinders the application of motion-language models. We propose a plug-and-play 2D motion interface that leverages cross-modal alignment and a real-video adapter to enable zero-shot adaptation of 3D-pretrained models to 2D inputs without architectural modifications. This approach effectively circumvents data bottlenecks by eliminating the need for model retraining. Evaluated on a newly constructed real-world video benchmark, our method achieves performance comparable to 3D-input baselines and significantly outperforms training from scratch. By overcoming limitations inherent to monocular video scenarios, this work establishes an efficient paradigm for the generalized deployment of motion understanding models, facilitating broader practical applications without reliance on scarce 3D motion data.

Abstract

Motion Language Models (MoLMs) typically understand human motions by tokenizing 3D motion and processing the resulting tokens using a language model. However, obtaining accurate 3D motions from monocular videos is challenging, limiting their real-world applicability. To address this issue, we introduce a plug-and-play 2D Motion Interface that enables 3D-pretrained MoLMs to accept 2D motion inputs without modifying or fine-tuning the original models. Experiments on public datasets show that our method achieves performance comparable to 3D motion inputs across multiple MoLMs and outperforms training MoLMs from scratch on 2D motions. We further construct a monocular real-world video motion evaluation dataset and introduce a real-video adapter, demonstrating the usefulness of 2D motions over 3D motions under the evaluated monocular pose-estimation setting. These results suggest that 2D motion provides a practical interface for deploying MoLMs in real-world motion understanding settings. Code is available at https://github.com/irajisamurai/2D-Motion-Interface.

Problem

• Motion Language Models
• 2D Motion Interface
• Monocular Video
• Real-world Applicability

Innovation

• Plug-and-Play 2D Motion Interface
• Motion Language Models
• Real-World Video Adapter
• Monocular Pose Estimation