LangLoc: "Tell Me What You See"

2026-07-06

AI Summary

This study addresses the challenge of fine-grained 2D pose (position and orientation) localization within known indoor 3D environments based solely on natural language descriptions, overcoming the limitations of prior methods that support only coarse scene retrieval. The authors propose LangLoc, a three-stage framework: first, a dual-branch GATv2 encoder integrates CLIP semantic features for scene retrieval; second, ray casting models object visibility to score a dense ground-plane grid and estimate an initial pose; third, a Bayesian dialogue module iteratively refines the pose posterior through active questioning to resolve ambiguities. This work presents the first end-to-end system mapping free-form text to fine-grained indoor poses and introduces the first large-scale indoor language localization benchmark with pose annotations, encompassing over 1,300 scenes and 13,000 descriptions. The method outperforms the previous state of the art by 8% in Top-1 scene retrieval recall and achieves a median pose estimation error of 0.95 meters.

Abstract

We tackle fine-grained indoor localization from natural language: given a free-form description of one's surroundings, estimate the observer's 2D position and heading within a known 3D environment. Language queries are lightweight, privacy-preserving, and need no camera - yet prior work stops at coarse scene retrieval and cannot resolve an intra-scene pose. We close this gap with LangLoc, a three-stage pipeline that (i) retrieves the correct scene via a dual-branch GATv2 encoder with CLIP semantic features, surpassing the previous best by 8 percentage points in Top-1 recall; (ii) estimates position and heading by scoring a dense floor grid through ray-cast object visibility, reaching a median error of 0.95 m; and (iii) resolves residual ambiguity through a Bayesian dialog module that asks targeted yes/no questions and updates a pose posterior until the location is pinpointed. To support this task we contribute a benchmark of $13{,}000{+}$ pose-indexed natural-language descriptions over $1{,}300{+}$ indoor 3D scans.

Problem

• fine-grained indoor localization
• natural language
• pose estimation
• 3D environment
• position and heading

Innovation

• fine-grained indoor localization
• natural language grounding
• ray-cast visibility
• Bayesian dialogue
• GATv2-CLIP fusion