PIGEON: VLM-Driven Object Navigation via Points of Interest Selection

2025-11-17

AI Summary

Achieving target-object navigation in unknown environments remains a core challenge for embodied intelligence, where existing approaches struggle to balance high-frequency decision-making with semantic foresight. This paper proposes a point-of-interest (POI)-guided visual-language-model (VLM)-driven navigation framework: it employs a lightweight semantic memory for rapid, fine-grained decisions; leverages a large-scale VLM (PIGEON-VL) to dynamically identify semantically salient POIs; integrates a dual-timescale (fast–slow) decision architecture; and incorporates RLVR (Reinforcement Learning with Verifiable Rewards) to generate high-quality, reward-verified RL data. The method requires no environmental priors or task-specific fine-tuning. Evaluated on standard object navigation benchmarks, it achieves state-of-the-art zero-shot transfer performance, significantly improving navigation efficiency, semantic consistency, and action coherence.

Technology Category

• Computer Vision: Language and Vision
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Navigating to a specified object in an unknown environment is a fundamental yet challenging capability of embodied intelligence. However, current methods struggle to balance decision frequency with intelligence, resulting in decisions lacking foresight or discontinuous actions. In this work, we propose PIGEON: Point of Interest Guided Exploration for Object Navigation with VLM, maintaining a lightweight and semantically aligned snapshot memory during exploration as semantic input for the exploration strategy. We use a large Visual-Language Model (VLM), named PIGEON-VL, to select Points of Interest (PoI) formed during exploration and then employ a lower-level planner for action output, increasing the decision frequency. Additionally, this PoI-based decision-making enables the generation of Reinforcement Learning with Verifiable Reward (RLVR) data suitable for simulators. Experiments on classic object navigation benchmarks demonstrate that our zero-shot transfer method achieves state-of-the-art performance, while RLVR further enhances the model's semantic guidance capabilities, enabling deep reasoning during real-time navigation.

Problem

• Balancing decision frequency with intelligence in object navigation
• Enabling foresightful decisions through Points of Interest selection
• Generating verifiable reinforcement learning data for simulators

Innovation

• Uses VLM to select Points of Interest
• Maintains lightweight semantic snapshot memory
• Generates verifiable reinforcement learning data