Hybrid User Interfaces: Past, Present, and Future of Complementary Cross-Device Interaction in Mixed Reality

2025-09-05

AI Summary

This paper addresses persistent challenges in the hybrid user interface (HUI) domain—namely, terminological inconsistency, conceptual ambiguity, and fragmented research—particularly in scenarios involving coordinated interaction between 2D devices and mixed reality (MR) environments, where a unified design framework and theoretical foundation remain absent. Through a structured literature review spanning nearly three decades of cross-device interaction, MR, and human–computer interaction (HCI) research, we propose the first integrative HUI taxonomy. This taxonomy rigorously defines the HUI design space, core challenges, and emerging opportunities. Crucially, it unifies device modalities, interaction coupling mechanisms, and contextual adaptability within a single analytical framework, thereby establishing a coherent theoretical foundation for HUIs. The resulting classification system provides practitioners and researchers with an actionable tool for designing multi-device collaborative interactions, along with principled guidance and consensus on future research directions.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Application Domains: Humanities & Computational Social Science
• Intelligent Robots: Human-Robot Interaction

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

We investigate hybrid user interfaces (HUIs), aiming to establish a cohesive understanding and adopt consistent terminology for this nascent research area. HUIs combine heterogeneous devices in complementary roles, leveraging the distinct benefits of each. Our work focuses on cross-device interaction between 2D devices and mixed reality environments, which are particularly compelling, leveraging the familiarity of traditional 2D platforms while providing spatial awareness and immersion. Although such HUIs have been prominently explored in the context of mixed reality by prior work, we still lack a cohesive understanding of the unique design possibilities and challenges of such combinations, resulting in a fragmented research landscape. We conducted a systematic survey and present a taxonomy of HUIs that combine conventional display technology and mixed reality environments. Based on this, we discuss past and current challenges, the evolution of definitions, and prospective opportunities to tie together the past 30 years of research with our vision of future HUIs.

Problem

• Establish cohesive understanding and terminology for hybrid user interfaces
• Combine 2D devices with mixed reality environments for complementary interaction
• Address fragmented research landscape and identify design challenges

Innovation

• Combining heterogeneous devices for complementary roles
• Leveraging 2D devices with mixed reality environments
• Systematic survey and taxonomy for cross-device interaction