InteRecon: Towards Reconstructing Interactivity of Personal Memorable Items in Mixed Reality

2025-02-14

AI Summary

Existing digitization methods (e.g., photography, 3D scanning) capture only the visual appearance of mementos, omitting their physical interactivity—thereby limiting the emotional depth of memory archiving. To address this, we propose Interactive Digital Items (IDIs), a novel paradigm that for the first time integrates three core dimensions of physical interactivity—geometric structure, manipulable interface, and embedded content—into a unified framework for digitizing personal memory objects. We develop the first augmented reality (AR) prototype system supporting user-driven annotation, content binding, and real-time physical behavior modeling, combining interactive 3D reconstruction with user-guided interface recognition. Empirical evaluation demonstrates that IDIs effectively reconstruct multimodal interactions of everyday objects. An exploratory user study confirms that IDIs significantly enhance memory retrieval depth and strengthen emotional connection to archived memories.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Studies of user behavior, including longitudinal effects of personalized systems
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Digital capturing of memorable personal items is a key way to archive personal memories. Although current digitization methods (e.g., photos, videos, 3D scanning) can replicate the physical appearance of an item, they often cannot preserve its real-world interactivity. We present Interactive Digital Item (IDI), a concept of reconstructing both the physical appearance and, more importantly, the interactivity of an item. We first conducted a formative study to understand users' expectations of IDI, identifying key physical interactivity features, including geometry, interfaces, and embedded content of items. Informed by these findings, we developed InteRecon, an AR prototype enabling personal reconstruction functions for IDI creation. An exploratory study was conducted to assess the feasibility of using InteRecon and explore the potential of IDI to enrich personal memory archives. Results show that InteRecon is feasible for IDI creation, and the concept of IDI brings new opportunities for augmenting personal memory archives.

Problem

• Reconstruct interactivity of personal items
• Preserve physical appearance and interactivity
• Augment personal memory archives digitally

Innovation

• Reconstructs item interactivity in AR
• Develops Interactive Digital Item concept
• Enables personal memory archive augmentation