Towards Understanding Camera Motions in Any Video

2025-04-21

AI Summary

This work addresses the lack of systematic evaluation and modeling for camera motion understanding in videos. We introduce CameraBench—the first large-scale benchmark grounded in professional filmmaking practice—and propose a cinematographer-informed primitive taxonomy for camera motion classification. Methodologically, we integrate multi-stage expert annotation, human-factor experiments, Structure-from-Motion (SfM)-based geometric analysis, and fine-tuned video-language models (VLMs) to establish a generative modeling paradigm that jointly leverages geometric perception and semantic understanding. Key contributions include: (1) empirical demonstration of domain expertise’s critical role in motion annotation quality; (2) substantial improvement in model discrimination among visually confusable motions (e.g., zoom vs. translation); and (3) state-of-the-art performance on motion-augmented captioning, video question answering, and cross-modal retrieval tasks.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We introduce CameraBench, a large-scale dataset and benchmark designed to assess and improve camera motion understanding. CameraBench consists of ~3,000 diverse internet videos, annotated by experts through a rigorous multi-stage quality control process. One of our contributions is a taxonomy of camera motion primitives, designed in collaboration with cinematographers. We find, for example, that some motions like"follow"(or tracking) require understanding scene content like moving subjects. We conduct a large-scale human study to quantify human annotation performance, revealing that domain expertise and tutorial-based training can significantly enhance accuracy. For example, a novice may confuse zoom-in (a change of intrinsics) with translating forward (a change of extrinsics), but can be trained to differentiate the two. Using CameraBench, we evaluate Structure-from-Motion (SfM) and Video-Language Models (VLMs), finding that SfM models struggle to capture semantic primitives that depend on scene content, while VLMs struggle to capture geometric primitives that require precise estimation of trajectories. We then fine-tune a generative VLM on CameraBench to achieve the best of both worlds and showcase its applications, including motion-augmented captioning, video question answering, and video-text retrieval. We hope our taxonomy, benchmark, and tutorials will drive future efforts towards the ultimate goal of understanding camera motions in any video.

Problem

• Assessing camera motion understanding in diverse videos
• Differentiating geometric and semantic camera motion primitives
• Improving model performance on motion-related video tasks

Innovation

• Large-scale dataset with expert annotations
• Taxonomy of camera motion primitives
• Fine-tuned generative VLM for dual capabilities