Natural Language Camera Movement Understanding

2026-07-03

AI Summary

This work addresses the longstanding challenge that existing vision-language models (VLMs) struggle to accurately interpret camera motion described in natural language, often conflating translation with rotation, left–right directions, and camera versus object motion. To tackle this, the study formally establishes camera motion understanding as a distinct task, introducing a two-tier cinematographic taxonomy and constructing the first atomic-level evaluation benchmark comprising both real-world and synthetic videos. The authors further propose a multi-source data augmentation strategy to train a VLM-8B model. After fine-tuning, the model outperforms Gemini 3.1 Pro by 10% on real videos and 11% on synthetic ones, yet remains substantially below human performance, thereby laying a systematic foundation for future research in this domain.

Abstract

Understanding camera movement in natural language is critical for training and evaluating video generation models, among other applications. However, we demonstrate that existing vision-language models (VLMs) fail this task in surprising ways, frequently confusing translation with rotation, left with right, and object movement with camera movement. To address these limitations, we establish natural language camera movement understanding as a standalone research task. We introduce a two-level cinematographic taxonomy and an extensive, atomic benchmark featuring both real and synthetic videos. Furthermore, we curate a large-scale, multi-source training set enhanced by targeted camera movement augmentation. Our fine-tuned VLM-8B outperforms Gemini 3.1 Pro by 10% and 11% on our benchmark's real and synthetic videos, respectively. Despite these gains, a significant gap remains relative to human performance, underscoring the need to promote and facilitate future research on natural language camera movement understanding.

Problem

• camera movement
• natural language understanding
• vision-language models
• video generation
• cinematographic understanding

Innovation

• camera movement understanding
• vision-language models
• cinematographic taxonomy
• video benchmark
• data augmentation