Say it or AI it: Evaluating Hands-Free Text Correction in Virtual Reality

2025-12-12

AI Summary

To address the challenge of controller-free text correction in virtual reality (VR), this paper proposes a voice-agnostic, fully AI-driven, hands-free correction framework leveraging large language models (LLMs) for end-to-end text rectification. Unlike conventional voice-input-based approaches, our work is the first to systematically evaluate the feasibility and usability of LLMs for controller-free text correction in VR environments. We integrate the method into a VR interaction framework and conduct user studies measuring System Usability Scale (SUS) scores, task completion rates, and correction latency. Results demonstrate statistically significant improvements over voice input: SUS score increases by 18.3 points, task success rate rises by 12.7%, and average correction time decreases by 24.1%. The core contribution lies in empirically validating LLMs as a highly usable, low-latency paradigm for hands-free text correction in VR—establishing a novel, practical foundation for immersive, controller-free text interaction.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Text entry in Virtual Reality (VR) is challenging, even when accounting for the use of controllers. Prior work has tackled this challenge head-on, improving the efficiency of input methods. These techniques have the advantage of allowing for relatively straightforward text correction. However, text correction without the use of controllers is a topic that has not received the same amount of attention, even though it can be desirable in several scenarios, and can even be the source of frustration. Large language models have been adopted and evaluated as a corrective methodology, given their high power for predictions. Nevertheless, their predictions are not always correct, which can lead to lower usability. In this paper, we investigate whether, for text correction in VR that is hands-free, the use of AI could surpass in terms of usability and efficiency. We observed better usability for AI text correction when compared to voice input.

Problem

• Evaluating hands-free text correction methods in VR
• Comparing AI-based correction with voice input for usability
• Addressing lack of controller-free text correction in VR

Innovation

• AI text correction surpasses voice input usability
• Hands-free VR text correction using large language models
• Evaluating AI predictions for efficiency in virtual reality