Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies

2026-07-12

AI Summary

This work addresses the limitations of existing reinforcement learning (RL) approaches for robot navigation, which lack semantic reasoning capabilities, and the high computational cost of vision-language models (VLMs), which hinders real-time deployment. To bridge this gap, the authors propose HUMA, a hybrid architecture that dynamically coordinates RL and VLM inference: an efficient RL policy handles routine navigation, while the VLM is selectively activated only when humans enter sensitive regions to perform contextual reasoning. This design achieves both real-time performance and socially aware behavior, yielding significant improvements in task success rates—by 20% on Social-MP3D and 3% on Social-HM3D—while substantially reducing personal space violations and human-robot collisions. The practical feasibility of HUMA is further validated through real-world deployment on the Mirokaï robotic platform.

Abstract

As mobile robots become more integrated into everyday human environments, social robot navigation is becoming essential for ensuring human comfort, safety, and trust. While reinforcement learning (RL) navigation policies provide the fast inference and reactive behavior necessary for real-time deployment, they still lack flexible semantic reasoning capabilities and often fail to generalize to complex social scenarios. Recent approaches have increasingly turned to vision-language models (VLMs) in place of RL policies to improve semantic and social reasoning in robot navigation. Nevertheless, their high computational cost and slow inference remain major barriers to real-time deployment. To overcome these limitations, we introduce HUMA (Hybrid Understanding for Multi-modal social Navigation), a hybrid architecture that dynamically balances the computational efficiency of RL policies with the deep semantic understanding of VLMs. Our approach uses a reactive RL policy to handle low-density, routine navigation tasks, while conditioning it on a post-trained high-level VLM when a human enters sensitive situations, such as the robot's proximity zone. We evaluate HUMA on the Social-MP3D and Social-HM3D benchmarks, where it achieves task success improvements of 20% and 3%, respectively, while significantly reducing personal space violations and human collisions against state-of-the-art baselines. Extensive ablation studies validate each architectural component, and real-world deployment on the Mirokaï mobile robot further demonstrates the practical viability of our approach.

Problem

• social robot navigation
• reinforcement learning
• vision-language models
• semantic reasoning
• real-time deployment

Innovation

• Conditional VLM Reasoning
• Hybrid Architecture
• Social Navigation
• Reinforcement Learning
• Vision-Language Models