ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Models

2026-06-26

AI Summary

This work proposes a general-purpose audio embedding framework that addresses the limitations of existing audio-text retrieval models, which are typically optimized solely for caption matching and struggle to support diverse objectives or controllable retrieval. For the first time, natural language instructions are integrated into the embedding process, leveraging a pretrained large audio-language model to transfer its capabilities in audio understanding, instruction following, and reasoning. The approach employs a contrastive dual-encoder architecture and utilizes large-scale multimodal pretraining to facilitate embedding space transfer learning. Experimental results demonstrate that the model achieves strong performance on standard audio and speech retrieval benchmarks while exhibiting remarkable compositional generalization and instruction-controllable retrieval abilities, underscoring its potential as a universal audio embedding model.

Abstract

Recent advances in language--audio retrieval have been largely driven by contrastive dual-encoder architectures that align audio and text in a shared embedding space. While effective, existing retrieval embeddings are primarily optimized for audio--caption matching, limiting their ability to support diverse retrieval objectives and controllable retrieval behaviors. We present ALM2Vec, a universal audio embedding framework derived from pretrained large audio--language models (LALMs). By transferring the audio understanding, instruction-following, and reasoning capabilities acquired through large-scale multimodal training, ALM2Vec learns a unified embedding space for retrieval across audio domains and task types. Beyond conventional text--audio retrieval, ALM2Vec incorporates natural-language instructions into the embedding process, enabling instruction-aware retrieval for scenarios such as audio question answering and aspect-conditioned retrieval. Experimental results show that ALM2Vec achieves competitive performance on standard audio and speech retrieval benchmarks while exhibiting promising compositional and controllable retrieval capabilities, highlighting its potential as a unified audio embedding model for retrieval across domains, tasks, and user intents.

Problem

• audio-language retrieval
• universal audio retrieval
• controllable retrieval
• instruction-aware retrieval
• audio embeddings

Innovation

• universal audio embedding
• large audio-language models
• instruction-aware retrieval
• compositional retrieval
• controllable audio retrieval