A Browser-Based Gesture-Driven Avatar Interaction Framework for Metaverse Onboarding Environments
This work addresses the limitations of existing metaverse systems, which predominantly rely on handheld controllers or keyboards and thus struggle to deliver natural, controller-free immersive interaction within web browsers. The paper proposes a lightweight, purely web-deployable, controller-free interaction framework that, for the first time, integrates real-time hand and head gesture recognition based on MediaPipe with two controller-free locomotion techniques—arm-swinging navigation and in-place walking. This integration enables users to drive their avatars and interact naturally with multimedia content, documents, and quizzes through intuitive body movements. A structured evaluation with five participants demonstrates the system’s feasibility in standard browser environments and reveals critical design trade-offs between interaction accuracy and sense of immersion across different input modalities.