Rethinking Implicit Spatial Representation in Visuomotor Policy Learning

2026-06-13

AI Summary

This work addresses the limitations of existing visuomotor policies, which suffer from the loss of fine-grained spatial information due to repeated downsampling in encoders—particularly under low-resolution observations—and the unclear efficacy of implicit spatial representations such as spatial Softmax. The study systematically analyzes the impact of pooling methods on robotic manipulation tasks, revealing that spatial Softmax yields compact and stable implicit representations. Building on this insight, the authors propose PRISM, a novel encoder that introduces a top-down, multi-scale cross-attention mechanism to integrate fine-grained spatial details while preserving high-level semantics. With only a 15.4% increase in parameters, PRISM consistently improves performance across multiple tasks and policy backbones, notably boosting success rates on the low-resolution ToolHang task from 5.0% to 13.4%, thereby establishing a new paradigm for efficient visual policy representation.

Abstract

Generative model-based imitation learning has become a widely adopted paradigm for robotic manipulation, where policy performance depends critically on the conditioned visual representations. Although spatial softmax-based representations have been adopted in prior visuomotor policies, their effectiveness and underlying mechanisms remain insufficiently understood. This work rethinks the use of spatial softmax pooling: do such implicit spatial representations provide effective and stable visual features for robotic manipulation? Through systematic studies of different pooling methods in visual encoders, we find that this pooling operation produces compact and stable spatial representations, which outperform feature-value representations, despite using substantially fewer dimensions. Complementary saliency analysis further suggests that these spatial representations guide the encoder to focus more consistently on task-relevant regions. However, this advantage is limited by a representation bottleneck in current visual encoders: repeated downsampling operations weaken fine-grained spatial information before the action-generation module can use it, especially under low-resolution observations. Motivated by these findings, we propose PRISM, a visual encoder that preserves multiscale implicit spatial information through top-down cross-attention fusion. Experiments across multiple tasks and policy backbones show consistent improvements. In particular, on the low-resolution, high-precision ToolHang task, PRISM shows clear gains, improving the average success rate from 5.0% to 13.4% while increasing parameters by only 15.4%. These results support the use of multiscale implicit spatial representations as an effective and efficient design principle for robotic manipulation.

Problem

• implicit spatial representation
• visuomotor policy
• spatial softmax
• visual encoder
• robotic manipulation

Innovation

• implicit spatial representation
• spatial softmax
• multiscale fusion
• cross-attention
• visuomotor policy