Cambrian-P: Pose-Grounded Video Understanding

2026-05-21

AI Summary

This work addresses the limitation of existing video multimodal large language models, which typically disregard camera pose and treat video frames as isolated 2D images, thereby failing to capture spatial consistency in the real world. To overcome this, the study introduces camera pose as a fundamental supervisory signal for the first time, proposing learnable camera tokens and a pose regression head. The approach integrates pseudo-labeled pose training with a customized frame sampling strategy, unifying spatial awareness and general video understanding while enabling streaming pose estimation. Evaluated on VSI-Bench, the method achieves a 4.5–6.5% performance gain and demonstrates strong results across eight spatial and general video question-answering benchmarks. It also sets a new state-of-the-art in streaming pose estimation on ScanNet.

Abstract

Camera pose matters. The position and orientation of each viewpoint define a shared spatial coordinate frame that relates observations across video frames. Yet this signal is largely absent from multimodal LLMs (MLLMs) for video understanding, which process frames as isolated 2D snapshots, instead of the persistent scene humans perceive. We revisit pose as a lightweight supervisory signal and introduce Cambrian-P, a video MLLM augmented with per-frame learnable camera tokens and a pose regression head. With a carefully designed sampling scheme, the model achieves substantial gains of 4.5-6.5% on spatial reasoning benchmarks such as VSI-Bench, generalizes across eight additional spatial and general video QA benchmarks, and, as a byproduct, achieves state of the art streaming pose estimation on ScanNet. Surprisingly, training on pseudo-annotated poses from in-the-wild video further improves general video QA benchmarks, showing pose helps beyond spatial reasoning. Together, these results position camera pose as a fundamental signal for video models that reason about the physical world.

Problem

• camera pose
• video understanding
• spatial reasoning
• multimodal LLMs
• 3D scene perception

Innovation

• camera pose
• video understanding
• multimodal LLM
• spatial reasoning
• pose regression