Motion-as-Prompt: Enhancing Motion Reasoning in Multimodal Large Language Models via Motion-Guided Cross-Frame Visual Prompting

2026-08-12

AI Summary

Existing multimodal large language models struggle with video reasoning due to the loss of critical motion information caused by sparse uniform frame sampling, hindering accurate understanding of object movement, collisions, and causal interactions. To address this limitation, this work proposes Motion-as-Prompt (MaP), a novel framework that explicitly encodes cross-frame motion cues by directly injecting dense point trajectories as visual prompts into the input images. Notably, MaP operates without modifying the model architecture or requiring additional training, instead rendering dynamic information explicit within otherwise static sampled frames through trajectory-guided visual annotations. This approach effectively enhances the motion perception capabilities of frozen models. Experimental results demonstrate that MaP improves motion reasoning accuracy of GPT-5.5 by 4.2% on CLEVRER and 8.9% on Something-Something-v2, while preserving performance on non-motion-related tasks.

Abstract

Motion-centric video reasoning is fundamental to interactive applications such as robotic manipulation and autonomous navigation. However, multimodal large language models (MLLMs) typically process videos through sparse uniform sampling to control visual-token and attention costs. This strategy may discard critical transitions between sampled frames, limiting reasoning about object movement, collisions, and causal interactions. To mitigate this issue, we propose Motion-as-Prompt (MaP), a track-guided cross-frame visual prompting framework. MaP recovers dense point trajectories, selects motion-informative frames, and marks the trajectories accumulated between consecutive sampled frames directly onto the visual inputs, making otherwise hidden displacement, direction changes, and interactions observable to frozen MLLMs. Experiments on CLEVRER and Something-Something-v2 show that MaP consistently improves average motion-reasoning accuracy, yielding gains of 4.2% and 8.9% for GPT-5.5, respectively. Notably, these improvements are obtained without degrading non-motion understanding, highlighting the robustness of MaP. These results demonstrate that MaP provides a simple and effective solution for enhancing motion-centric video reasoning without model training or architectural modification. Project page:https://github.com/SunVictor23/MaP.

Problem

• motion reasoning
• multimodal large language models
• video understanding
• frame sampling
• visual prompting

Innovation

• Motion-as-Prompt
• multimodal large language models
• motion reasoning
• visual prompting
• trajectory annotation