Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation

2026-08-11

AI Summary

Existing methods for video camera motion understanding provide only clip-level holistic labels, making it difficult to model temporally varying and concurrently occurring composite motions within a shot. To address this limitation, this work reframes the task as a temporal localization and compositional recognition problem, requiring models to identify temporally coherent intervals and recognize all active motion types within them. We introduce CamChoreo, the first benchmark dataset enabling such fine-grained annotations, and propose CamDistill, a lightweight geometric knowledge distillation approach that fuses multimodal large language models, frozen 3D foundation model features, and knowledge distillation to generate efficient camera motion tokens. Experiments demonstrate that CamDistill achieves accuracy comparable to direct feature injection from expensive 3D teacher models—without executing them at inference—thereby advancing camera motion understanding toward fine-grained, temporally compositional recognition.

Abstract

Understanding camera motion is fundamental to video perception, with applications in spatial intelligence and controllable video generation. Multimodal large language models (MLLMs) provide a natural interface for this task, but existing work typically assigns one or more labels to an entire clip. Such clip-level recognition overlooks two defining properties of real camera motion: it can change within a shot, and multiple movements can occur simultaneously. We therefore formulate camera-motion understanding as temporally grounded, compositional recognition, which requires a model to localize motion-consistent intervals and identify every movement active within each interval. We introduce CamChoreo, a benchmark of 4,229 real single-shot clips with expert-annotated temporal segments. Its annotations use a compact vocabulary of 20 direction-aware labels, and nearly half of the segments contain compound camera motion, with multiple movement primitives active simultaneously. Recognizing such fine-grained, compositional motion is hard for current MLLMs, whose visual encoders emphasize semantic content rather than the geometric evidence on which camera motion depends. Directly injecting features from a frozen 3D foundation model addresses this gap, but requires running the expensive geometry model on every input; we refer to this baseline as CamInject. We instead propose CamDistill, which distills the same geometric knowledge into lightweight camera tokens during training and removes the 3D model at inference. CamDistill matches the accuracy of direct feature injection without running the 3D teacher at inference. Together, CamChoreo and CamDistill advance camera-motion understanding from clip-level labeling to temporally grounded, compositional recognition. Project page: https://ddz16.github.io/cammotion.github.io/.

Problem

• camera motion
• temporal grounding
• compositional recognition
• video perception
• multimodal large language models

Innovation

• temporally grounded
• compositional recognition
• geometric knowledge distillation
• camera motion understanding
• multimodal large language models