ReSiReg: Towards Spatially Consistent Semantics in Language-Conditioned Robotic Tasks

2026-06-17

AI Summary

This work addresses the limitations of existing vision-language models (VLMs) in robotic tasks, where dense embeddings often suffer from high noise and poor spatial consistency, hindering joint reasoning over semantics and 3D spatial relationships. The authors propose ReSiReg, a novel method that leverages intermediate VLM features for spatially consistent reconstruction. By clustering visual features into prototypes and associating them with language descriptions, ReSiReg reconstructs image patches as soft mixtures of prototype-level language embeddings, enabling joint optimization of semantic content and spatial structure. The resulting lightweight dense VLM contains only 25M parameters—significantly smaller than ViT-B yet achieving comparable performance—and demonstrates consistent improvements on OVSS and 3D scene mapping benchmarks. Real-world robotic experiments further validate its effectiveness, showing more coherent and accurate target activation regions.

Abstract

Vision-Language Models (VLMs) enable robots to follow open-language instructions. However, dense VLM embeddings have shown to be noisy and lack spatial consistency. This is problematic for robotic applications, which require simultaneous reasoning over semantics and 3D space. We examine spatial structure across recent VLMs and propose ReSiReg, a feature reconstruction method that uses spatially consistent VLM intermediates to improve dense language-grounded retrieval. ReSiReg clusters intermediates into visual prototypes, derives their language descriptors, and reconstructs each patch as a soft mixture of prototype-level language embeddings. We evaluate quantitatively on OVSS and 3D mapping across backbones, and qualitatively in real-world manipulation scenes. Quantitative results show improved dense retrieval; manipulation scenes show more spatially consistent target activations. We further provide a compact 25M dense VLM for robotic applications, substantially smaller than and competitive with ViT-B baselines. Available at https://resireg.github.io

Problem

• Vision-Language Models
• spatial consistency
• dense retrieval
• robotic tasks
• language-conditioned

Innovation

• spatial consistency
• vision-language models
• feature reconstruction
• dense retrieval
• robotic manipulation