2024 NASA SUITS Report: LLM-Driven Immersive Augmented Reality User Interface for Robotics and Space Exploration

2025-07-01

AI Summary

To address the machine perception bottleneck in augmented reality (AR) human–machine interaction under complex dynamic environments in deep-space exploration (e.g., Artemis missions)—particularly the challenge of 3D object pose estimation without ground-truth sensors—this paper proposes a large language model (LLM)-driven, non-intrusive immersive AR interaction system. Methodologically, it integrates ZED2 binocular vision, digital twin–based localization, and a Transformer architecture to develop DTTDNet, a lightweight, deeply fused 6-degree-of-freedom (6DoF) pose estimation algorithm. It further introduces LMCC, a localized task console enabling voice-driven LLM instruction parsing and real-time robotic control. Experimental evaluation in lunar- and Martian-simulated environments demonstrates centimeter-level localization accuracy and millisecond-scale response latency, with stable operation achieved without external ground-truth sensors. This work establishes the first end-to-end, scalable “perception–understanding–control” digital twin AR interaction paradigm for aerospace and industrial applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As modern computing advances, new interaction paradigms have emerged, particularly in Augmented Reality (AR), which overlays virtual interfaces onto physical objects. This evolution poses challenges in machine perception, especially for tasks like 3D object pose estimation in complex, dynamic environments. Our project addresses critical issues in human-robot interaction within mobile AR, focusing on non-intrusive, spatially aware interfaces. We present URSA, an LLM-driven immersive AR system developed for NASA's 2023-2024 SUITS challenge, targeting future spaceflight needs such as the Artemis missions. URSA integrates three core technologies: a head-mounted AR device (e.g., HoloLens) for intuitive visual feedback, voice control powered by large language models for hands-free interaction, and robot tracking algorithms that enable accurate 3D localization in dynamic settings. To enhance precision, we leverage digital twin localization technologies, using datasets like DTTD-Mobile and specialized hardware such as the ZED2 camera for real-world tracking under noise and occlusion. Our system enables real-time robot control and monitoring via an AR interface, even in the absence of ground-truth sensors--vital for hazardous or remote operations. Key contributions include: (1) a non-intrusive AR interface with LLM-based voice input; (2) a ZED2-based dataset tailored for non-rigid robotic bodies; (3) a Local Mission Control Console (LMCC) for mission visualization; (4) a transformer-based 6DoF pose estimator (DTTDNet) optimized for depth fusion and real-time tracking; and (5) end-to-end integration for astronaut mission support. This work advances digital twin applications in robotics, offering scalable solutions for both aerospace and industrial domains.

Problem

• Enhancing human-robot interaction in mobile AR for space exploration
• Improving 3D object pose estimation in dynamic environments
• Developing hands-free AR interfaces with LLM-driven voice control

Innovation

• LLM-driven AR system for space exploration
• Head-mounted AR with voice control
• Digital twin localization for 3D tracking