DeformTune: A Deformable XAI Music Prototype for Non-Musicians

2025-07-31

AI Summary

Current AI-based music generation tools rely on text prompts, professional interfaces, or instrument-like controls, imposing high cognitive and skill barriers for novices. To address this, we propose a deformable haptic interaction system tailored for non-expert users, integrating explainable AI (XAI) with multimodal, progressive guidance. Our approach leverages the MeasureVAE model to enable semantically grounded gesture-to-music mapping, while physical shape-changing interfaces and real-time visual feedback enhance users’ comprehension and controllability over the generative process. The key contribution is the first synergistic integration of deformable hardware, embodied interaction, and XAI for accessible music creation—significantly reducing cognitive load. Preliminary user studies demonstrate efficient task completion and identify critical design dimensions: clarity of control-to-music mapping, expressive granularity, and intensity of dynamic guidance.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Intelligent Robots: Embodied AI

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Many existing AI music generation tools rely on text prompts, complex interfaces, or instrument-like controls, which may require musical or technical knowledge that non-musicians do not possess. This paper introduces DeformTune, a prototype system that combines a tactile deformable interface with the MeasureVAE model to explore more intuitive, embodied, and explainable AI interaction. We conducted a preliminary study with 11 adult participants without formal musical training to investigate their experience with AI-assisted music creation. Thematic analysis of their feedback revealed recurring challenge--including unclear control mappings, limited expressive range, and the need for guidance throughout use. We discuss several design opportunities for enhancing explainability of AI, including multimodal feedback and progressive interaction support. These findings contribute early insights toward making AI music systems more explainable and empowering for novice users.

Problem

• Intuitive AI music creation for non-musicians
• Explainable interaction with deformable interface
• Reducing musical/technical barriers in AI tools

Innovation

• Deformable tactile interface for intuitive control
• MeasureVAE model for AI music generation
• Multimodal feedback enhances explainability