SRLM: Human-in-Loop Interactive Social Robot Navigation with Large Language Model and Deep Reinforcement Learning

2024-03-22

arXiv.org

AI Summary

Navigating social robots in high-density, dynamic pedestrian environments requires jointly addressing complex natural language instruction understanding, global semantic planning, and socially compliant motion control. Method: This paper proposes a closed-loop framework integrating large language models (LLMs) and deep reinforcement learning (DRL), featuring a novel Large Navigation Model (LNM) fused with a DRL planner and augmented by a Large Feedback Model (LFM) to correct LLM output biases. The architecture enables multimodal social context encoding, real-time instruction parsing, and human-in-the-loop feedback integration. Contribution/Results: Evaluated across multiple high-density scenarios, the approach achieves significant improvements over pure-DRL and end-to-end LLM baselines: +23.6% navigation success rate, +31.4% social compliance, and +27.9% instruction-following accuracy. It establishes the first unified navigation paradigm that is language-driven, socially aware, and motion-controllable.

Technology Category

• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

An interactive social robotic assistant must provide services in complex and crowded spaces while adapting its behavior based on real-time human language commands or feedback. In this paper, we propose a novel hybrid approach called Social Robot Planner (SRLM), which integrates Large Language Models (LLM) and Deep Reinforcement Learning (DRL) to navigate through human-filled public spaces and provide multiple social services. SRLM infers global planning from human-in-loop commands in real-time, and encodes social information into a LLM-based large navigation model (LNM) for low-level motion execution. Moreover, a DRL-based planner is designed to maintain benchmarking performance, which is blended with LNM by a large feedback model (LFM) to address the instability of current text and LLM-driven LNM. Finally, SRLM demonstrates outstanding performance in extensive experiments. More details about this work are available at: https://sites.google.com/view/navi-srlm

Problem

• Human-Robot Interaction
• Natural Language Understanding
• Autonomous Navigation

Innovation

• SRLM System
• Large Language Model
• Smart Learning Technology