Libretto: Giving LLM Agents a Sense of Musical Structure

2026-06-21

AI Summary

Existing music generation systems lack structured scrutiny of audio outputs and controllable editing capabilities. This work proposes Libretto, a novel framework that, for the first time, translates symbolic music into a structured, editable, and quantifiable representation amenable to language models. By introducing a symbolic syntax explicitly modeling note onsets, voice assignments, and measure structures tailored for large language models (LLMs), Libretto establishes a multidimensional statistical evaluation space encompassing rhythm, harmony, and melody. The framework enables multitask generation, retrieval-based diagnostics, copyright risk mitigation, and iterative self-refinement along musically meaningful structural axes. It significantly enhances the structural coherence and controllability of generated outputs across diverse tasks, including inpainting, reference-guided composition, progressive transformation, and educational music generation.

Abstract

Generative music systems can now produce impressive audio from text prompts, but audio outputs are difficult to inspect, edit, and diagnose as musical structure. We introduce Libretto, an agent-facing framework for symbolic music generation and revision. Libretto uses an LLM-native grammar with explicit onset slots, voices, and bar-level organization, then evaluates each piece in a corpus-calibrated statistical space over rhythm, harmony, melody, texture, form, and variation. The same structural axes support retrieval, diagnosis, copy-risk control, and iterative self-revision. Across gap filling, reference-guided full-piece generation, gradual morphing, and educational music generation, Libretto turns symbolic music from a raw token sequence into a measurable and editable object for language-model agents.

Problem

• generative music
• musical structure
• symbolic music
• audio editing
• music diagnosis

Innovation

• symbolic music generation
• LLM-native grammar
• music structure representation
• self-revision
• corpus-calibrated evaluation