Problem
Research questions and friction points this paper is trying to address.
Text-to-music
performance benchmarking
heterogeneous metrics
system architecture
musical-content alignment
TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking
🤖 AI Summary
为解决文本转音乐系统性能评估难题，TTM-Bench框架通过统一协议衡量音乐内容一致性及计算效率，提供可解释的评估指标。
📝 Abstract
Text-to-music (TTM) systems are increasingly used to generate musical audio from natural-language descriptions. Robust evaluation is therefore essential, yet reliable performance comparison remains challenging. This difficulty stems from differences in system architecture, supported conditioning information, and access mode, as well as heterogeneous and fragmented metrics that cannot be applied uniformly across systems. To address these challenges, we introduce TTM-Bench, a framework that defines a common protocol for systematic, reproducible performance benchmarking of contemporary TTM systems. It evaluates performance along two dimensions: musical-content alignment, quantified by interpretable semantic, genre, and musical-descriptor agreement scores against a common musical specification and summarized by an aggregate score; and computational efficiency, characterized by generation latency and real-time factor, alongside resource use for local models and cost for hosted services. We demonstrate the framework through a preliminary comparative case study, illustrating the complementary evidence captured by these dimensions. The results show that higher musical-content alignment does not systematically coincide with lower computational demands, highlighting the importance of assessing TTM performance through distinct, interpretable measures rather than a reductive overall indicator.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Text-to-Music
Performance Benchmarking
Musical-Content Alignment
Computational Efficiency
🔎 Similar Papers
No similar papers found.
