From Language to Navigation Goals: A Vision-Language Approach for Semantic Navigation of Mobile Robots Using RGB-D Perception

2026-07-15

AI Summary

This work addresses the challenge of enabling non-expert users to control mobile robot navigation through natural language commands. It proposes a modular, ROS 2–based language-driven navigation framework that integrates natural language understanding, RGB-D semantic perception, and Nav2 autonomous navigation to achieve end-to-end mapping from linguistic instructions to navigational goals. The system supports context-aware instruction parsing and cross-platform deployment. Validated on both TurtleBot3 Waffle and Unitree Go2 platforms, it accurately identifies linguistically referenced targets, estimates their spatial locations, generates feasible navigation paths, and provides natural language feedback. This approach significantly enhances the intuitiveness and robustness of human–robot interaction in real-world environments.

Abstract

Natural language interaction provides an intuitive way for non-expert users to communicate with robotic platforms. However, transforming user requests into executable navigation actions remains a challenging task, requiring the integration of language understanding, environment perception, and autonomous navigation. This work presents a language-driven navigation framework that enables mobile robots to interpret user requests in natural language to move the robot to a destination and autonomously navigate towards it. The framework is composed of modular ROS 2 components that cooperate to transform natural language instructions into navigation actions. Given a natural language request referring to a target in the environment (e.g., "go to the mail box"), the system identifies the referenced object, estimates its position using RGB-D data, and generates a navigation goal, which is then executed through the ROS 2 Nav2 navigation stack. The ROS 2-based implementation facilitates portability across different robotic platforms, requiring only the configuration of the corresponding topics and services. The system is evaluated in both simulation and real-world scenarios using a TurtleBot3 Waffle and a Unitree Go2 robot with a RealSense camera. Experimental results show that the framework successfully interprets both direct commands and contextual requests, generates meaningful natural-language feedback, and navigates towards the desired target. These results demonstrate the feasibility of combining semantic perception and autonomous navigation to provide an intuitive human-robot interaction paradigm. Code will be released as open source upon acceptance.

Problem

• semantic navigation
• natural language interaction
• mobile robots
• RGB-D perception
• navigation goals

Innovation

• vision-language navigation
• semantic navigation
• natural language interaction
• RGB-D perception
• ROS 2 modular framework