World Mouse: Exploring Interactions with a Cross-Reality Cursor

2026-03-11

AI Summary

Existing extended reality (XR) systems face limitations in natural interaction modalities such as touch and gaze, which often compromise precision, accessibility, or user comfort, thereby hindering efficient fine-grained manipulation within complex 3D mixed-reality environments. To address this, this work proposes World Mouse—a cross-reality cursor system that extends the familiar 2D desktop mouse paradigm into 3D physical-virtual spaces. By leveraging semantic segmentation and mesh reconstruction, the system transforms real-world objects into interactive surfaces, enabling precise cursor control through surface normal-based positioning and seamless inter-object navigation via spatial interpolation. World Mouse overcomes the conventional confinement of cursors to purely virtual domains, supporting smooth transitions from screen-based to real-world interactions. Prototype evaluations demonstrate its effectiveness in performing delicate manipulation and cross-domain tasks, validating its feasibility and advantages in mixed reality scenarios.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Computer Vision: 3D Computer Vision

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

As Extended Reality (XR) systems increasingly map and understand the physical world, interacting with these blended representations remains challenging. The current push for "natural" inputs has its trade-offs: touch is limited by human reach and fatigue, while gaze often lacks the precision for fine interaction. To bridge this gap, we introduce World Mouse, a cross-reality cursor that reinterprets the familiar 2D desktop mouse for complex 3D scenes. The system is driven by two core mechanisms: within-object interaction, which uses surface normals for precise cursor placement, and between-object navigation, which leverages interpolation to traverse empty space. Unlike previous virtual-only approaches, World Mouse leverages semantic segmentation and mesh reconstruction to treat physical objects as interactive surfaces. Through a series of prototypes, including object manipulation and screen-to-world transitions, we illustrate how cross-reality cursors may enable seamless interactions across real and virtual environments.

Problem

• Extended Reality
• cross-reality interaction
• 3D interaction
• cursor
• human-computer interaction

Innovation

• cross-reality cursor
• semantic segmentation
• mesh reconstruction
• within-object interaction
• between-object navigation