🤖 AI Summary
This work addresses the limitations of existing metaverse systems, which predominantly rely on handheld controllers or keyboards and thus struggle to deliver natural, controller-free immersive interaction within web browsers. The paper proposes a lightweight, purely web-deployable, controller-free interaction framework that, for the first time, integrates real-time hand and head gesture recognition based on MediaPipe with two controller-free locomotion techniques—arm-swinging navigation and in-place walking. This integration enables users to drive their avatars and interact naturally with multimedia content, documents, and quizzes through intuitive body movements. A structured evaluation with five participants demonstrates the system’s feasibility in standard browser environments and reveals critical design trade-offs between interaction accuracy and sense of immersion across different input modalities.
📝 Abstract
Avatar interaction shapes how engaging and immersive a metaverse experience feels, and for that interaction to feel natural, avatars need to respond to users without forcing them through a controller-based interface first. This paper describes a gesture-driven interaction layer built for a browser-based metaverse onboarding environment, where users explore a set of virtual rooms as an avatar and interact with embedded video, document, and quiz content using hand, arm, and head gestures instead of a keyboard or controller. The system combines real-time gesture recognition (Google MediaPipe) with two alternative locomotion techniques - hand-raise navigation and in-place walking - so users can trade off precision against physical immersion depending on the task. The contribution is the integration, deployment, and evaluation of these techniques as a single lightweight, web-deployable, controller-free interaction model, assessed through a structured internal onboarding session with five participants. We report what worked, what didn't, and the design trade-offs that came out of combining these techniques in one deployed system.