Studies for : A Human-AI Co-Creative Sound Artwork Using a Real-time Multi-channel Sound Generation Model

2025-10-29

AI Summary

Sound art archives are typically static, hindering stylistic evolution and long-term artistic continuity. Method: This study introduces the “living archive” paradigm—replacing static preservation with AI-driven dynamic sound generation. We propose SpecMaskGIT, a lightweight, real-time eight-channel generative model trained on over 200 hours of Evala’s historical works. The framework integrates artist-in-the-loop feedback, personalized fine-tuning, and controllable stochasticity to enable human–AI co-creation. Contribution/Results: Deployed in a three-month immersive installation, the system continuously generated acoustically coherent and stylistically novel audio content aligned with the artist’s aesthetic. This constitutes the first empirical demonstration of AI’s capacity to sustain artistic style, revitalize sonic heritage, and support collaborative sound art creation—validating both technical feasibility and aesthetic viability.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

This paper explores the integration of AI technologies into the artistic workflow through the creation of Studies for, a generative sound installation developed in collaboration with sound artist Evala (https://www.ntticc.or.jp/en/archive/works/studies-for/). The installation employs SpecMaskGIT, a lightweight yet high-quality sound generation AI model, to generate and playback eight-channel sound in real-time, creating an immersive auditory experience over the course of a three-month exhibition. The work is grounded in the concept of a "new form of archive," which aims to preserve the artistic style of an artist while expanding beyond artists' past artworks by continued generation of new sound elements. This speculative approach to archival preservation is facilitated by training the AI model on a dataset consisting of over 200 hours of Evala's past sound artworks. By addressing key requirements in the co-creation of art using AI, this study highlights the value of the following aspects: (1) the necessity of integrating artist feedback, (2) datasets derived from an artist's past works, and (3) ensuring the inclusion of unexpected, novel outputs. In Studies for, the model was designed to reflect the artist's artistic identity while generating new, previously unheard sounds, making it a fitting realization of the concept of "a new form of archive." We propose a Human-AI co-creation framework for effectively incorporating sound generation AI models into the sound art creation process and suggest new possibilities for creating and archiving sound art that extend an artist's work beyond their physical existence. Demo page: https://sony.github.io/studies-for/

Problem

• Developing real-time multi-channel AI sound generation for immersive art
• Creating AI archives preserving artistic style while generating novel elements
• Establishing human-AI co-creation framework integrating artist feedback and datasets

Innovation

• Real-time multi-channel sound generation using SpecMaskGIT model
• Training AI on artist's past works for style preservation
• Human-AI co-creative framework integrating artist feedback