LIME: Learning Intent-aware Camera Motion from Egocentric Video

2026-07-02

AI Summary

Existing methods struggle to generate appropriate camera motions aligned with natural language intent. This work addresses this challenge by formulating language-guided camera motion as a sequence of discrete actions and introduces a novel approach that self-supervises the extraction of multi-intent camera motion signals from first-person videos. The proposed model jointly predicts both the content to be revealed in the next view and multiple hypothesized target viewpoints. It employs an architecture combining autoregressive observation gain with a continuous flow-matching pose head, trained on SE(3) relative poses conditioned on language descriptions. By learning from passive human video data, the method acquires active perception strategies and successfully generates camera motions that conform to linguistic instructions across diverse robotic tasks.

Abstract

Autonomous robots often need to move their camera before they can act: to inspect an object, reveal an occluded region, or obtain a view that responds to a user's intent. While vision-language navigation translates instructions to base motion and vision-language-action policies map instructions to manipulation actions, language-conditioned camera motion remains comparatively underexplored as a first-class action. We formulate language-conditioned camera motion generation: given a current RGB observation and a free-form natural-language intent, predict a relative target camera pose for the next observation. This task is inherently non-trivial: viewpoint changes are driven by latent perceptual intentions, and a valid motion may operate at different semantic granularity, from entering a room to looking around a corner, inspecting a visible object, or revealing an occluded detail. To model this structure, we mine multi-intention camera-motion supervision from egocentric video, pairing plausible intents and observation-gain descriptions with relative SE(3) target poses. We propose LIME, a vision-language camera-motion generator that combines an auto-regressive observation-gain output with a continuous flow-matching pose head. This design lets the model jointly predict what the next view should reveal while representing multi-hypothesis target views. Across experiments and downstream robotic tasks, we show that LIME can learn to actively choose camera poses from passive human video, turning ordinary egocentric recordings into supervision for intent-aware active perception.

Problem

• language-conditioned camera motion
• intent-aware perception
• egocentric video
• active perception
• camera pose prediction

Innovation

• language-conditioned camera motion
• egocentric video
• intent-aware perception
• flow-matching pose estimation
• observation-gain modeling