GeoProp: Grounding Robot State in Vision for Generalist Manipulation

2026-07-08

AI Summary

Existing embodied intelligence methods often underperform pure vision-based policies due to the lack of explicit alignment between proprioceptive signals and visual features, hindering effective fusion of robot state and scene understanding. This work proposes GeoProp, a lightweight, plug-and-play adapter that explicitly aligns proprioception with vision through geometric projection: it maps proprioceptive states onto the image plane, samples local visual features to construct state tokens, and injects spatial priors via FiLM modulation. Additionally, GeoProp incorporates short-term motion-predicted coordinates to enable intention-driven, forward-looking feature sampling. This approach achieves the first explicit geometric alignment between proprioception and visual features, improving Diffusion Policy by 8.7% across 63 simulated tasks, pi₀ by 4.0% on RoboTwin, and delivering an average 10.6% performance gain in real-world settings—all with only a 2–3% increase in model parameters.

Abstract

Proprioception is fundamental to robotic manipulation, yet standard fusion methods often treat it as an isolated vector lacking explicit alignment with visual tokens. Without a direct correspondence between 3D kinematics and 2D feature maps, manipulation policies struggle to ground the robot's state within the scene, frequently underperforming even vision-only baselines. To address this, we introduce GeoProp, a lightweight, plug-and-play adapter that aligns proprioception with vision through explicit geometric grounding and spatial feature sampling. GeoProp projects the robot state onto the image plane to sample localized visual features, constructing a grounded state token. It then injects state-derived spatial priors into the corresponding visual features via FiLM modulation. To capture motion intent, GeoProp further samples features at a short-horizon predicted coordinate derived from recent kinematics, providing look-ahead visual context. Across 67 tasks, GeoProp improves Diffusion Policy by 8.7% on 63 simulation tasks and pi_0 by 4.0% on the RoboTwin subset, and yields a 10.6% average gain across both policy families in the real world, while adding only 2-3% to the parameter count. These results demonstrate that GeoProp is a simple yet high-impact inductive bias for generalist embodied policies. Project page: https://alibaba-damo-academy.github.io/GeoProp/.

Problem

• proprioception
• vision-language grounding
• robotic manipulation
• state-visual alignment
• embodied AI

Innovation

• geometric grounding
• proprioception-vision fusion
• spatial feature sampling
• FiLM modulation
• look-ahead visual context