Understanding the Use of a Large Language Model-Powered Guide to Make Virtual Reality Accessible for Blind and Low Vision People

2026-03-10

AI Summary

This work addresses the significant accessibility challenges posed by the growing prevalence of social virtual reality (Social VR) for blind and low-vision users. We propose and evaluate an AI-powered voice navigation assistant driven by a large language model (LLM) to support navigation and social interaction within immersive VR environments. Through a user study involving 16 blind and low-vision participants, we reveal— for the first time—the assistant’s dual role as both a functional tool and a social companion in Social VR, alongside diverse user interaction strategies. Our findings demonstrate the effectiveness of LLM-based assistance in enhancing Social VR accessibility and offer critical design insights for future AI-augmented systems tailored to visually impaired users.

Technology Category

• Humans and AI: AI for Accessibility
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As social virtual reality (VR) grows more popular, addressing accessibility for blind and low vision (BLV) users is increasingly critical. Researchers have proposed an AI "sighted guide" to help users navigate VR and answer their questions, but it has not been studied with users. To address this gap, we developed a large language model (LLM)-powered guide and studied its use with 16 BLV participants in virtual environments with confederates posing as other users. We found that when alone, participants treated the guide as a tool, but treated it companionably around others, giving it nicknames, rationalizing its mistakes with its appearance, and encouraging confederate-guide interaction. Our work furthers understanding of guides as a versatile method for VR accessibility and presents design recommendations for future guides.

Problem

• virtual reality
• blind and low vision
• accessibility
• AI guide
• large language model

Innovation

• large language model
• virtual reality accessibility
• blind and low vision
• AI sighted guide
• social interaction