From Temporal to Spatial: Designing Spatialized Interactions with Segmented-audios in Immersive Environments for Active Engagement with Performing Arts Intangible Cultural Heritage

2025-05-23

AI Summary

Passive listening in intangible cultural heritage (ICH) transmission—particularly for performing arts such as Peking Opera—leads to low user engagement and superficial comprehension. Method: This study proposes a spatialized, interactive segmented-audio VR system that transforms linear audio into an explorable 3D soundscape. We introduce a novel t-SNE–driven audio semantic clustering and spatiotemporal mapping paradigm, integrating VR-based spatial navigation, fine-grained audio segmentation, and participatory co-design workshops. Through iterative design, we identify two new active listening interaction patterns: Progressive and Adaptive. Contribution/Results: A user study with 16 participants demonstrates statistically significant improvements in active listening depth (*p* < 0.01) and cultural understanding (+37.2%). The work establishes a reusable spatial-auditory interaction design framework for the living transmission of ICH, advancing immersive, user-centered approaches to heritage preservation.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Cognitive Modeling & Cognitive Systems: Affective Computing
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Security and Privacy: Data transparency and provenance

Abstract

Performance artforms like Peking opera face transmission challenges due to the extensive passive listening required to understand their nuance. To create engaging forms of experiencing auditory Intangible Cultural Heritage (ICH), we designed a spatial interaction-based segmented-audio (SISA) Virtual Reality system that transforms passive ICH experiences into active ones. We undertook: (1) a co-design workshop with seven stakeholders to establish design requirements, (2) prototyping with five participants to validate design elements, and (3) user testing with 16 participants exploring Peking Opera. We designed transformations of temporal music into spatial interactions by cutting sounds into short audio segments, applying t-SNE algorithm to cluster audio segments spatially. Users navigate through these sounds by their similarity in audio property. Analysis revealed two distinct interaction patterns (Progressive and Adaptive), and demonstrated SISA's efficacy in facilitating active auditory ICH engagement. Our work illuminates the design process for enriching traditional performance artform using spatially-tuned forms of listening.

Problem

• Transforming passive Peking opera listening into active engagement
• Designing spatial interactions with segmented-audios in VR
• Enhancing auditory Intangible Cultural Heritage through spatial clustering

Innovation

• Spatialized segmented-audio VR system
• t-SNE algorithm clusters audio spatially
• Progressive and Adaptive interaction patterns