musicolors: Bridging Sound and Visuals For Synesthetic Creative Musical Experience

2025-03-18

AI Summary

Music visualization research has long prioritized technical implementation over users’ authentic creative contexts, neglecting the synesthetic mechanisms underlying music–visual interaction. Method: This study proposes a synesthesia-centered design paradigm and develops *musicolors*—a lightweight, real-time, cross-platform web library for music visualization—supporting ideation sketching, system integration, and multimodal listening. Leveraging Web Audio API and Canvas/WebGL rendering within a modular frontend architecture, its effectiveness is validated through qualitative user studies (in-depth interviews and contextual observation). Contribution/Results: The work bridges a critical gap in everyday music–visual co-design, identifying three core values: inspiration generation, systemic interoperability, and perceptual expansion. It yields empirically grounded interaction design guidelines that shift music visualization from technical demonstration toward creative empowerment, enhancing musical ideation efficiency and cross-modal expressive capacity.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Computer Vision: Multi-modal Vision
• Humans and AI: Interaction Techniques and Devices

Application Category

• Web Mining and Content Analysis: Web data visualization
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Music visualization is an important medium that enables synesthetic experiences and creative inspiration. However, previous research focused mainly on the technical and theoretical aspects, overlooking users' everyday interaction with music visualizations. This gap highlights the pressing need for research on how music visualization influences users in synesthetic creative experiences and where they are heading. Thus, we developed musicolors, a web-based music visualization library available in real-time. Additionally, we conducted a qualitative user study with composers, developers, and listeners to explore how they use musicolors to appreciate and get inspiration and craft the music-visual interaction. The results show that musicolors provides a rich value of music visualization to users through sketching for musical ideas, integrating visualizations with other systems or platforms, and synesthetic listening. Based on these findings, we also provide guidelines for future music visualizations to offer a more interactive and creative experience.

Problem

• Exploring user interaction with music visualization in creative experiences.
• Developing a real-time web-based music visualization tool, musicolors.
• Investigating how musicolors enhances synesthetic and creative musical inspiration.

Innovation

• Real-time web-based music visualization library
• Qualitative user study with diverse participants
• Guidelines for interactive creative music visualizations