Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddings

2026-01-16

AI Summary

This work proposes LIVI, a novel approach to music cover identification that addresses the high computational cost and reliance on complex audio features in existing methods. LIVI is the first method to completely eliminate the need for lyric transcription during inference, instead leveraging supervision signals derived from lyric transcriptions and pretrained text embeddings solely during training to construct lyric-aligned audio embeddings. Through end-to-end audio representation learning and efficient similarity retrieval, LIVI achieves accuracy on par with or superior to state-of-the-art harmony-based methods across multiple benchmark datasets, while substantially reducing inference time and resource consumption. By decoupling lyric information from the inference pipeline yet preserving its supervisory benefit during training, the method overcomes limitations of conventional multimodal architectures, offering a highly accurate yet computationally efficient solution.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Music Cover Retrieval, also known as Version Identification, aims to recognize distinct renditions of the same underlying musical work, a task central to catalog management, copyright enforcement, and music retrieval. State-of-the-art approaches have largely focused on harmonic and melodic features, employing increasingly complex audio pipelines designed to be invariant to musical attributes that often vary widely across covers. While effective, these methods demand substantial training time and computational resources. By contrast, lyrics constitute a strong invariant across covers, though their use has been limited by the difficulty of extracting them accurately and efficiently from polyphonic audio. Early methods relied on simple frameworks that limited downstream performance, while more recent systems deliver stronger results but require large models integrated within complex multimodal architectures. We introduce LIVI (Lyrics-Informed Version Identification), an approach that seeks to balance retrieval accuracy with computational efficiency. First, LIVI leverages supervision from state-of-the-art transcription and text embedding models during training to achieve retrieval accuracy on par with--or superior to--harmonic-based systems. Second, LIVI remains lightweight and efficient by removing the transcription step at inference, challenging the dominance of complexity-heavy pipelines.

Problem

• Music Cover Retrieval
• Version Identification
• Lyrics Extraction
• Audio Embeddings
• Scalability

Innovation

• Lyrics-Aligned Audio Embeddings
• Music Cover Retrieval
• Version Identification
• Efficient Inference
• Supervised Representation Learning