Socially-Aware Robot Navigation Enhanced by Bidirectional Natural Language Conversations Using Large Language Models

2024-09-08

AI Summary

Traditional robot navigation prioritizes efficiency and obstacle avoidance while neglecting human behavior modeling and bidirectional interaction in dynamic shared spaces. To address this, we propose HSAC-LLM, a novel socially aware navigation framework that integrates large language models (LLMs) with deep reinforcement learning (DRL) to enable proactive, natural-language-based negotiation prior to potential collisions—facilitating intent alignment and collaborative avoidance. The framework incorporates a hybrid soft actor-critic (HSAC) algorithm, multimodal prompt engineering, and a real-time speech-to-text/text-to-speech interface. It is rigorously validated in both Gazebo simulation and real-world environments. Experimental results demonstrate that HSAC-LLM achieves a 37% improvement in interaction success rate, reduces average navigation time by 22%, and lowers collision frequency to 0.03 collisions per kilometer—outperforming state-of-the-art DRL baselines.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Human-Robot Interaction
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Robot navigation is crucial across various domains, yet traditional methods focus on efficiency and obstacle avoidance, often overlooking human behavior in shared spaces. With the rise of service robots, socially aware navigation has gained prominence. However, existing approaches primarily predict pedestrian movements or issue alerts, lacking true human-robot interaction. We introduce Hybrid Soft Actor-Critic with Large Language Model (HSAC-LLM), a novel framework for socially aware navigation. By integrating deep reinforcement learning with large language models, HSAC-LLM enables bidirectional natural language interactions, predicting both continuous and discrete navigation actions. When potential collisions arise, the robot proactively communicates with pedestrians to determine avoidance strategies. Experiments in 2D simulation, Gazebo, and real-world environments demonstrate that HSAC-LLM outperforms state-of-the-art DRL methods in interaction, navigation, and obstacle avoidance. This paradigm advances effective human-robot interactions in dynamic settings. Videos are available at https://hsacllm.github.io/.

Problem

• Enhancing robot navigation with human social awareness
• Enabling bidirectional human-robot natural language interactions
• Improving collision avoidance through proactive pedestrian communication

Innovation

• Integrates deep reinforcement learning with LLMs
• Enables bidirectional natural language interactions
• Proactively communicates for collision avoidance