ObjectReact: Learning Object-Relative Control for Visual Navigation

2025-09-11

AI Summary

Existing monocular vision-based navigation methods relying on topological maps employ “image-relative” control, which suffers from poor generalization due to sensitivity to viewpoint, ego-pose, and sensor configuration. Method: We propose an “object-relative” navigation paradigm that grounds control decisions in geometric and semantic relationships among scene objects. Our approach constructs a viewpoint-invariant 3D relative scene graph and generates a WayObject Costmap as input to a purely object-level local policy network—eliminating reliance on RGB inputs. Contribution/Results: This is the first method to decouple control prediction from image matching, enabling zero-shot generalization to unseen paths, bidirectional navigation, and cross-platform deployment. Evaluated across multi-task simulations and real-world environments, it significantly outperforms image-relative baselines. Notably, it achieves zero-shot sim-to-real transfer—demonstrating strong robustness and practical applicability without real-world fine-tuning.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Localization, Mapping, and Navigation
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Visual navigation using only a single camera and a topological map has recently become an appealing alternative to methods that require additional sensors and 3D maps. This is typically achieved through an "image-relative" approach to estimating control from a given pair of current observation and subgoal image. However, image-level representations of the world have limitations because images are strictly tied to the agent's pose and embodiment. In contrast, objects, being a property of the map, offer an embodiment- and trajectory-invariant world representation. In this work, we present a new paradigm of learning "object-relative" control that exhibits several desirable characteristics: a) new routes can be traversed without strictly requiring to imitate prior experience, b) the control prediction problem can be decoupled from solving the image matching problem, and c) high invariance can be achieved in cross-embodiment deployment for variations across both training-testing and mapping-execution settings. We propose a topometric map representation in the form of a "relative" 3D scene graph, which is used to obtain more informative object-level global path planning costs. We train a local controller, dubbed "ObjectReact", conditioned directly on a high-level "WayObject Costmap" representation that eliminates the need for an explicit RGB input. We demonstrate the advantages of learning object-relative control over its image-relative counterpart across sensor height variations and multiple navigation tasks that challenge the underlying spatial understanding capability, e.g., navigating a map trajectory in the reverse direction. We further show that our sim-only policy is able to generalize well to real-world indoor environments. Code and supplementary material are accessible via project page: https://object-react.github.io/

Problem

• Learning object-relative control for visual navigation tasks
• Overcoming limitations of image-level representations in navigation
• Enabling cross-embodiment deployment with invariant object representations

Innovation

• Object-relative control learning paradigm
• Topometric relative 3D scene graph
• WayObject Costmap local controller