Steering dense music retrieval with open-vocabulary concept discovery

2026-08-09

AI Summary

This work addresses the inaccuracy in controllable music editing caused by semantic misalignment between open-vocabulary text prompts and audio representations. To mitigate this issue, the authors propose a lightweight, training-free approach that formulates concept attribution as a sparse inverse problem for the first time. Specifically, within a shared multimodal embedding space, the method employs sparse inverse optimization to recover a set of audio features that align with the target textual concept while preserving the intrinsic geometric structure of the original audio. Notably, the approach requires neither paired audio-text supervision nor retraining of sparse autoencoders. It achieves superior precision in amplifying or suppressing specified concepts, striking a better balance between edit intensity and content preservation, thereby effectively minimizing the introduction of irrelevant features and the loss of critical musical information.

Abstract

Controllable music retrieval lets users find music that is, for example, more ambient, less distorted, or without guitar while preserving the other semantic content of an original seed query. Sparse autoencoders (SAEs) are a promising interface for this kind of concept-level control, but a key problem remains: given a free-form text concept, which sparse features should be edited? In shared multimodal embedding spaces, standard attribution methods often select neurons that match the concept's wording but not the audio examples that express it. This leads to weak or unstable edits: relevant features are missed when concepts are distributed across neurons, while others are selected due to text alignment rather than audio-side structure. We address this with a lightweight, training-free method that recovers a sparse set of audio features whose decoded representation reconstructs the target concept while remaining consistent with audio-space geometry. This reframes concept attribution as a sparse inversion problem rather than a text-side neuron-ranking heuristic. The method requires neither paired audio-text supervision nor SAE retraining. We evaluate this approach in steerable music retrieval and show that the recovered supports align more closely with concept-bearing audio examples and achieve a stronger trade-off between edit strength and preservation than alignment baselines, enabling more precise concept amplification and suppression with reduced drift on preservation metrics.

Problem

• controllable music retrieval
• open-vocabulary concept
• sparse autoencoders
• concept attribution
• multimodal embedding

Innovation

• sparse inversion
• controllable music retrieval
• open-vocabulary concept
• multimodal embedding
• sparse autoencoder