SIR: Structured Image Representations for Explainable Robot Learning

2026-06-29

AI Summary

Current vision-based robotic policies often lack explicit structure, rendering them susceptible to distractions and producing decisions that are difficult to interpret. This work proposes Structured Image Representation (SIR), which introduces a learnable, sparse scene graph as an intermediate representation within the policy for the first time. An end-to-end model extracts a task-relevant subgraph from a fully connected scene graph, enabling interpretable action decisions grounded in relational reasoning. Evaluated on RoboCasa, SIR achieves a success rate of 19.5%, significantly outperforming the image-based baseline at 14.81%. Moreover, subgraph analysis reveals how the model attends to distractors or overlooks critical objects, effectively exposing dataset biases and spurious correlations that undermine generalization.

Abstract

Existing robot policies based on learned visual embeddings lack explicit structure and are sensitive to visual distractions. Thus, the representations that drive their behaviour are often opaque, making their decision-making process difficult to interpret. To address this, we introduce Structured Image Representations (SIR), a method that leverages Scene Graphs (SGs) as an intermediate representation for robot policy learning. Our approach first constructs a fully connected graph, using image-derived features as initial node representations. Then, a module learns to sparsify this graph end-to-end, creating a task-relevant sub-graph that is passed to the action generation model. This process makes our model intrinsically explainable. Evaluations on RoboCasa show that our sparse graph policies outperform image-based baselines on average with 19.5% vs 14.81% success rate. Most importantly, we show that the learned sparse graphs are a powerful tool for model analysis. By analysing when the model's sub-graph deviates from human expectation, such as by including distractor nodes or omitting key objects, we successfully uncover dataset biases, including spurious correlations and positional biases. https://github.com/intuitive-robots/SIR_Model

Problem

• explainable robot learning
• structured representations
• visual distractions
• opaque decision-making
• scene understanding

Innovation

• Structured Image Representations
• Scene Graphs
• Explainable Robot Learning
• Graph Sparsification
• Visual Reasoning