SSI-Policy: Learning Structured Scene Interfaces for Vision-Language Robotic Manipulation

2026-06-25

AI Summary

This work addresses the challenge of coordinating spatial localization, task perception, and precise control in robotic manipulation under low-data conditions. The authors propose SSI-Policy, a novel framework featuring a Structured Scene Interface (SSI) that operates solely on RGB inputs. SSI unifies monocular depth features, language-anchored object layouts, and instruction-conditioned 2D motion trajectories into a shared representation, effectively decoupling perception from control. Its modular architecture integrates depth estimation, vision–language alignment, and trajectory prediction to construct a joint geometric and motion representation, enabling training without action-labeled videos and facilitating cross-embodiment transfer. Evaluated on the LIBERO benchmark, SSI-Policy surpasses the strongest baseline by nearly 15% with only ten demonstrations per task and demonstrates superior spatial reasoning, cross-embodiment generalization, and contact-rich manipulation across 13 real-world tasks.

Abstract

Real-world robotic manipulation demands spatial grounding, task-aware reasoning, and precise control. Learning such capabilities becomes particularly challenging in the low-data regime. Prior methods often trade off scalable task-level reasoning and explicit physical structure: video-based approaches can drift geometrically over long horizons, 3D approaches often require depth sensing, and many flow/trajectory interfaces emphasize motion without an explicit RGB-only geometric representation. We introduce SSI-Policy, a modular framework built around a Structured Scene Interface (SSI) -- a unified, RGB-only intermediate representation that jointly encodes monocular depth features, language-grounded object layouts, and instruction-conditioned 2D motion trajectories. Critically, SSI is robot-agnostic and trainable from action-free video, decoupling perception from control so that the downstream policy can learn from few demonstrations. On the LIBERO benchmark with only 10 demonstrations per task, SSI-Policy improves over the strongest prior method by nearly 15\% and remains competitive with 50-demo methods that leverage large-scale external pretraining. Ablations show that geometric and motion cues provide complementary benefits within the shared interface. We further validate on 13 real-world tasks spanning spatial reasoning, cross-embodiment transfer, and contact-rich manipulation.

Problem

• vision-language manipulation
• low-data regime
• spatial grounding
• RGB-only representation
• robotic manipulation

Innovation

• Structured Scene Interface
• RGB-only representation
• vision-language manipulation
• few-shot robotic learning
• perception-control decoupling