STORM: Slot-based Task-aware Object-centric Representation for robotic Manipulation

2026-01-28

AI Summary

This work addresses the limited robustness and composability of vision foundation models in robotic manipulation tasks due to their lack of explicit object-level structure. To overcome this, the authors propose STORM—a lightweight, object-centric adapter module that enhances frozen vision foundation models through a semantic-aware slot mechanism. STORM employs a multi-stage training strategy: first, it performs vision-language pretraining guided by language embeddings to align visual and semantic representations; then, it jointly fine-tunes downstream manipulation policies while keeping the large model frozen. This approach efficiently constructs task-aware, object-centric representations without retraining the foundation model, effectively mitigating slot collapse and preserving semantic consistency. Experiments demonstrate that STORM significantly outperforms both direct use of frozen features and end-to-end learned object representations in object discovery and simulated manipulation tasks, achieving superior generalization under perturbations and enhanced control performance.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Visual foundation models provide strong perceptual features for robotics, but their dense representations lack explicit object-level structure, limiting robustness and contractility in manipulation tasks. We propose STORM (Slot-based Task-aware Object-centric Representation for robotic Manipulation), a lightweight object-centric adaptation module that augments frozen visual foundation models with a small set of semantic-aware slots for robotic manipulation. Rather than retraining large backbones, STORM employs a multi-phase training strategy: object-centric slots are first stabilized through visual--semantic pretraining using language embeddings, then jointly adapted with a downstream manipulation policy. This staged learning prevents degenerate slot formation and preserves semantic consistency while aligning perception with task objectives. Experiments on object discovery benchmarks and simulated manipulation tasks show that STORM improves generalization to visual distractors, and control performance compared to directly using frozen foundation model features or training object-centric representations end-to-end. Our results highlight multi-phase adaptation as an efficient mechanism for transforming generic foundation model features into task-aware object-centric representations for robotic control.

Problem

• object-centric representation
• visual foundation models
• robotic manipulation
• semantic structure
• dense representations

Innovation

• slot-based representation
• task-aware adaptation
• object-centric perception
• visual foundation models
• multi-phase training