SongSage: A Large Musical Language Model with Lyric Generative Pre-training

2026-01-03

arXiv.org

AI Summary

This work addresses the limitations of general-purpose large language models in lyric comprehension and semantic playlist modeling by introducing SongSage, the first lyric-centric music language model. SongSage is pretrained on LyricBank, a corpus of 5.48 billion tokens, and further fine-tuned with 775,000 multi-task instructions. The study contributes a novel lyric-oriented music language model, the PlaylistSense evaluation benchmark, and demonstrates significant performance gains over strong baselines across nine tasks—including lyric generation, rewriting, and playlist understanding—while maintaining competitive general-domain capabilities on MMLU. Notably, SongSage supports zero-shot lyric continuation and playlist recommendation, showcasing its effectiveness in both specialized music understanding and broader language tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models have achieved significant success in various domains, yet their understanding of lyric-centric knowledge has not been fully explored. In this work, we first introduce PlaylistSense, a dataset to evaluate the playlist understanding capability of language models. PlaylistSense encompasses ten types of user queries derived from common real-world perspectives, challenging LLMs to accurately grasp playlist features and address diverse user intents. Comprehensive evaluations indicate that current general-purpose LLMs still have potential for improvement in playlist understanding. Inspired by this, we introduce SongSage, a large musical language model equipped with diverse lyric-centric intelligence through lyric generative pretraining. SongSage undergoes continual pretraining on LyricBank, a carefully curated corpus of 5.48 billion tokens focused on lyrical content, followed by fine-tuning with LyricBank-SFT, a meticulously crafted instruction set comprising 775k samples across nine core lyric-centric tasks. Experimental results demonstrate that SongSage exhibits a strong understanding of lyric-centric knowledge, excels in rewriting user queries for zero-shot playlist recommendations, generates and continues lyrics effectively, and performs proficiently across seven additional capabilities. Beyond its lyric-centric expertise, SongSage also retains general knowledge comprehension and achieves a competitive MMLU score. We will keep the datasets inaccessible due to copyright restrictions and release the SongSage and training script to ensure reproducibility and support music AI research and applications, the datasets release plan details are provided in the appendix.

Problem

• lyric-centric knowledge
• playlist understanding
• large language models
• music AI
• user intent

Innovation

• lyric generative pre-training
• large musical language model
• LyricBank
• PlaylistSense
• instruction fine-tuning