Seek to Segment: Active Perception for Panoramic Referring Segmentation

2026-07-02

AI Summary

This work addresses the limitation of existing referring expression segmentation methods, which are confined to static images and thus ill-suited for embodied agents requiring active perception in 360° environments. To bridge this gap, the paper introduces the Active Panoramic Referring Expression Segmentation (APRS) task and presents PanoSeeker, an agent that integrates a vision-language model with an explicit EgoSphere spatial memory mechanism to enable efficient, non-redundant active viewpoint exploration and target segmentation. By jointly optimizing the search policy through supervised fine-tuning and reinforcement learning, the proposed approach achieves state-of-the-art performance on a newly established APRS benchmark, significantly outperforming existing methods in both search efficiency and segmentation accuracy.

Abstract

Existing referring segmentation models passively process static images captured from fixed perspectives, limiting their applicability in Embodied AI, where agents must perform active perception in the continuous 360$^\circ$ environments. To bridge this gap, we introduce a novel task: Active Panoramic Referring Segmentation (APRS). In this setting, an agent is required to adjust its viewing direction ($Δθ, Δφ$) to explore the 360$^\circ$ environment, seeking the object specified by a user instruction for segmentation. To tackle this challenging task, we propose PanoSeeker, a memory-augmented agent for efficient APRS. Rather than relying on heuristic scanning, PanoSeeker integrates a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory. By progressively integrating sequential local observations into a unified 360$^\circ$ representation, EgoSphere enables the agent to plan efficient and non-redundant search trajectories. Once the target is found, the agent performs active viewpoint alignment and outputs the segmentation mask. Furthermore, we curate an expert-annotated search trajectory dataset with memory timelines for Supervised Fine-Tuning, followed by Reinforcement Learning post-training to explicitly optimize PanoSeeker's exploration efficiency. Extensive experiments on our newly established APRS benchmark demonstrate that PanoSeeker achieves superior search efficiency and segmentation accuracy, significantly outperforming adapted state-of-the-art baselines.

Problem

• Active Perception
• Panoramic Referring Segmentation
• Embodied AI
• 360-degree Environment
• Viewpoint Planning

Innovation

• Active Perception
• Panoramic Referring Segmentation
• EgoSphere
• Vision-Language Model
• Embodied AI