Predicting Timbre Traits for Interpretable Assessment of Musical Sound Synthesizers

2026-06-29

AI Summary

This study addresses the limited interpretability of existing audio synthesizer evaluation metrics—such as Fréchet Audio Distance (FAD)—which fail to capture timbral characteristics in a human-understandable manner, thereby hindering targeted model improvements. To bridge this gap, the authors propose a timbre attribute prediction model leveraging CLAP pre-trained audio embeddings combined with a shallow learnable module. Trained on the RWC instrument database and human ratings across 20 semantic timbre dimensions, the approach uniquely integrates interpretable timbral semantics into neural audio synthesis evaluation. The model preserves distribution-level performance by reproducing FAD rankings while enabling qualitative, single-sample diagnostic insights. Predictions exhibit strong correlation with human judgments (r = 0.66, p < 0.001), allowing precise identification of specific timbral attributes requiring refinement.

Abstract

Measuring neural audio synthesizers' performance is now routinely conducted using distribution based metrics such as the Fréchet Audio Distance (FAD). Although this metric can be correlated with human perception, it offers limited interpretability beyond ranking different approaches. In this paper, we introduce a deep neural timbre trait predictor composed of a pretrained audio neural embedding (CLAP), and a shallow learnable component. The latter is trained using the RWC musical instrument database and human judgments of 20 timbre descriptions (e.g., woody, percussive, rumbling, etc.) for 31 instruments. The resulting model shows strong correlation with average human ratings (r = 0.66, p < 0.001). We then demonstrate the benefit of this predictor for evaluating the performance of TokenSynth, a neural sound synthesizer. First, the Mean Absolute Error (MAE) computed over the set of generated sounds under different conditioning modalities of the model provides the same ranking as a FAD computed with the RWC database as a reference, suggesting that the proposed predictors are able to provide equivalent information on a distributional basis. Second, because the model is able to qualitatively analyze isolated sounds, we can determine which generated sounds could be improved and identify specific timbral dimensions that need adjustment.

Problem

• timbre traits
• interpretable evaluation
• neural audio synthesizers
• Fréchet Audio Distance
• sound synthesis assessment

Innovation

• timbre trait prediction
• interpretable evaluation
• neural audio synthesis
• CLAP embedding
• human-aligned audio metrics