Mixer Metaphors: audio interfaces for non-musical applications

2025-04-16

AI Summary

This work investigates whether audio interface metaphors—such as analog synthesizers and mixing consoles—can effectively transfer to embodied control of large language models (LLMs). To this end, we designed and implemented a physical interactive hardware prototype enabling artists to manipulate semantic content, stylistic attributes, and structural properties of LLM outputs in real time via knobs, faders, and other tactile controls. This represents the first systematic application of audio-domain embodied interaction paradigms to non-musical AI systems. Integrating low-latency LLM API calls and conducting a one-week comparative user study, we found that the audio-metaphor interface significantly enhances operational directness, frequency of improvisational exploration, and creative immersion relative to conventional software-only interfaces; users reported stronger perceived parameter agency and capacity for creative intervention. Our core contribution is empirical validation that cross-modal metaphor supports more embodied, intelligible, and creatively generative interaction with complex AI systems.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Embodied AI

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The NIME conference traditionally focuses on interfaces for music and musical expression. In this paper we reverse this tradition to ask, can interfaces developed for music be successfully appropriated to non-musical applications? To help answer this question we designed and developed a new device, which uses interface metaphors borrowed from analogue synthesisers and audio mixing to physically control the intangible aspects of a Large Language Model. We compared two versions of the device, with and without the audio-inspired augmentations, with a group of artists who used each version over a one week period. Our results show that the use of audio-like controls afforded more immediate, direct and embodied control over the LLM, allowing users to creatively experiment and play with the device over its non-mixer counterpart. Our project demonstrates how cross-sensory metaphors can support creative thinking and embodied practice when designing new technological interfaces.

Problem

• Can music interfaces work for non-musical applications?
• How audio metaphors improve control over Large Language Models?
• Do cross-sensory metaphors enhance creative interface design?

Innovation

• Audio-inspired controls for LLM manipulation
• Cross-sensory metaphors enhance creative interfaces
• Mixer metaphors enable embodied AI interaction