NavAI: A Generalizable LLM Framework for Navigation Tasks in Virtual Reality Environments

2025-11-16

2025 40th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW)

AI Summary

Existing navigation methods struggle to support the diverse and complex goal-oriented tasks required in immersive virtual reality (VR) environments. To address this limitation, this work proposes NavAI—the first general-purpose VR navigation framework powered by a large language model (LLM). NavAI leverages the LLM as its core reasoning engine and integrates with VR interaction interfaces to enable end-to-end mapping from natural language instructions to navigational actions. The framework unifies the handling of both goal-directed and exploratory tasks and demonstrates strong cross-environment generalization. Experimental evaluation across three heterogeneous VR environments shows that NavAI achieves a success rate of 89% on goal-oriented tasks, confirming its effectiveness and versatility.

Technology Category

• Computer Vision: Language and Vision
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Navigation is one of the fundamental tasks for automated exploration in Virtual Reality (VR). Existing technologies primarily focus on path optimization in 360-degree image datasets and 3D simulators, which cannot be directly applied to immersive VR environments. To address this gap, we present navAI, a generalizable large language model (LLM)based navigation framework that supports both basic actions and complex goal-directed tasks across diverse VR applications. We evaluate NavAI in three distinct VR environments through goal-oriented and exploratory tasks. Results show that it achieves high accuracy, with an 89% success rate in goal-oriented tasks. Our analysis also highlights current limitations of relying entirely on LLMs, particularly in scenarios that require dynamic goal assessment. Finally, we discuss the limitations observed during the experiments and offer insights for future research directions.

Problem

• navigation
• virtual reality
• large language model
• generalizable framework
• immersive environments

Innovation

• Large Language Model
• Virtual Reality Navigation
• Generalizable Framework
• Goal-directed Task
• Immersive Environment