HCSG: Human-Centric Semantic-Geometric Reasoning for Vision-Language Navigation

2026-05-13

AI Summary

This work addresses a critical limitation in existing vision-and-language navigation (VLN) approaches, which treat pedestrians in dynamic indoor environments merely as moving obstacles while neglecting human intentions and social norms. To overcome this, the authors propose the Human-Centric Semantic-Geometric (HCSG) framework, which introduces a unified human understanding module that jointly models geometric trajectory prediction and semantic intention inference to construct a semantic-geometric joint representation. This representation is integrated into a topological map to support instruction-guided path planning. By incorporating a novel social distance loss, the system shifts from passive obstacle avoidance to proactive comprehension of human behavior while adhering to social conventions. Evaluated on the HA-VLNCE benchmark, the method achieves a 14% absolute improvement in success rate and reduces collision rate by 34%, substantially outperforming current state-of-the-art approaches.

Abstract

VLN has achieved remarkable progress by scaling data and model capacity. However, the assumption of a static environment breaks down in real-world indoor scenarios, where robots inevitably encounter dynamic pedestrians. Existing human-aware approaches typically treat humans merely as moving obstacles based on implicit visual cues, lacking the explicit reasoning required to interpret human intentions or maintain social norms. To address this, we propose HCSG, the first human-centric framework for VLN. This framework provides a robust foundation for safe, socially intelligent navigation in dynamic human-robot environments that shifts the paradigm from passive collision avoidance to active human behavior understanding. Specifically, HCSG introduces a unified Human Understanding Module that synergizes two key capabilities: (i) geometric forecasting, which predicts human pose and trajectory to anticipate future motion dynamics; and (ii) semantic interpretation, which leverages a Vision-Language Model (VLM) to generate natural language descriptions of human actions and intentions. These semantic-geometric representations are fused into the agent's topological map for instruction-conditioned planning. Furthermore, a social distance loss is introduced to enforce socially compliant interaction distances. Extensive experiments on the HA-VLNCE benchmark demonstrate that HCSG significantly outperforms state-of-the-art methods, achieving a 14% improvement in Success Rate and a 34% reduction in Collision Rate. Our project can be seen at https://haoxuanxu1024.github.io/HCSG/.

Problem

• Vision-Language Navigation
• Human-Aware Navigation
• Dynamic Environments
• Social Norms
• Human Intention Understanding

Innovation

• human-centric navigation
• semantic-geometric reasoning
• vision-language model
• socially compliant interaction
• trajectory forecasting