fARfetch: Enabling Collocated AR-HRC in Large Visually Diverse Environments with VLM-Driven AR Content Adaptation

2026-06-23

AI Summary

This work addresses the challenges of poor legibility of distant virtual content and difficulties in off-axis interaction that commonly hinder augmented reality (AR)-mediated human–robot collaboration in large-scale outdoor environments. The authors propose an AR adaptive system integrating semantic mapping, a miniature world view, and a vision-language model (VLM) to dynamically optimize the color, size, and orientation of AR content for sustained readability. This approach enables shared semantic mapping and context-aware interaction between AR headsets and quadrupedal robots. User studies demonstrate that the proposed method reduces task completion time by 66% and significantly lowers mental workload (43%), time pressure (34%), and frustration (66%), thereby substantially enhancing collaborative efficiency and user experience in complex outdoor scenarios.

Abstract

Augmented Reality (AR) can improve collocated human-robot collaboration by making robot state and intent visible and enabling intuitive control, yet large, visually diverse environments like the outdoors challenge both interaction and content legibility, especially at long distances and beyond visual line of sight. We present fARfetch, an AR-HRC system that integrates (i) shared semantic environment mapping across an AR headset and robot that visualizes detected landmarks in AR to support landmark-grounded go-to commands, (ii) a context-aware world-in-miniature representation of the shared environment for fine-grained path authoring, and (iii) vision-language-model driven AR view management that jointly adapts virtual content color, size, and orientation to maintain legibility in large visually diverse environments. We implement fARfetch with a Meta Quest 3 headset and Unitree Go2 quadruped robot, and conduct a within-subjects user study (N=13) on a real-world large-scale (30.5m) outdoor inspection task. fARfetch yielded significantly faster completion times than a non-AR baseline (66%) and significantly lower workload in mental demand (-43%), temporal demand (-34%), and frustration (-66%). A custom legibility survey indicated fARfetch effectively maintained virtual content legibility in the large outdoor environment.

Problem

• Augmented Reality
• Human-Robot Collaboration
• Content Legibility
• Visually Diverse Environments
• Outdoor AR

Innovation

• VLM-driven AR adaptation
• shared semantic mapping
• world-in-miniature interface
• collocated AR-HRC
• content legibility