Survey on the Evaluation of Generative Models in Music

2025-06-05

AI Summary

Existing music generation systems lack a systematic, interdisciplinary evaluation framework. Method: This paper introduces the first comprehensive assessment framework integrating musicological, engineering, and human-computer interaction perspectives, jointly evaluating output quality and model usability. It synthesizes subjective listening tests, objective audio metrics (e.g., Frechet Audio Distance, KL divergence), music-theoretic analysis, and user interaction studies to empirically compare over 30 evaluation methods. A taxonomy is proposed to clarify trade-offs among fidelity, diversity, interpretability, and practicality. Contribution/Results: The work unifies cross-disciplinary evaluation paradigms, delineates method-specific applicability boundaries, and establishes a reusable, principled methodology for scientifically benchmarking music generation models—advancing both reproducibility and rigor in generative music research.

Technology Category

• Natural Language Processing: Generation
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Research on generative systems in music has seen considerable attention and growth in recent years. A variety of attempts have been made to systematically evaluate such systems. We provide an interdisciplinary review of the common evaluation targets, methodologies, and metrics for the evaluation of both system output and model usability, covering subjective and objective approaches, qualitative and quantitative approaches, as well as empirical and computational methods. We discuss the advantages and challenges of such approaches from a musicological, an engineering, and an HCI perspective.

Problem

• Evaluating generative music models' output and usability
• Reviewing interdisciplinary evaluation methods and metrics
• Assessing approaches from musicology, engineering, and HCI

Innovation

• Interdisciplinary review of evaluation targets
• Subjective and objective evaluation methodologies
• Musicological, engineering, and HCI perspectives