Do Joint Language-Audio Embeddings Encode Perceptual Timbre Semantics?

2025-10-15

AI Summary

Prior work lacks systematic evaluation of whether joint language–audio embedding models effectively encode human-perceived timbral semantics (e.g., brightness, roughness, warmth). Method: We introduce the first psychoacoustically aligned benchmark to quantitatively assess cross-modal timbral representation capabilities of MS-CLAP, LAION-CLAP, and MuQ-MuLan on instrumental and sound-effect audio. Leveraging human-annotated timbral attributes as supervisory signals, we combine multimodal embedding analysis with perceptual dimension mapping to measure modeling fidelity of timbral semantics in text–audio alignment. Results: LAION-CLAP achieves superior and most consistent alignment across all perceptual dimensions—significantly outperforming competing models—demonstrating that its shared embedding space more faithfully reflects the structure of human timbre perception. This work establishes a novel, interpretable benchmark for audio semantic modeling and provides empirical grounding for perceptually grounded representation learning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Understanding and modeling the relationship between language and sound is critical for applications such as music information retrieval,text-guided music generation, and audio captioning. Central to these tasks is the use of joint language-audio embedding spaces, which map textual descriptions and auditory content into a shared embedding space. While multimodal embedding models such as MS-CLAP, LAION-CLAP, and MuQ-MuLan have shown strong performance in aligning language and audio, their correspondence to human perception of timbre, a multifaceted attribute encompassing qualities such as brightness, roughness, and warmth, remains underexplored. In this paper, we evaluate the above three joint language-audio embedding models on their ability to capture perceptual dimensions of timbre. Our findings show that LAION-CLAP consistently provides the most reliable alignment with human-perceived timbre semantics across both instrumental sounds and audio effects.

Problem

• Evaluating joint language-audio embeddings for timbre semantics
• Assessing alignment of multimodal models with human perception
• Comparing embedding models on perceptual timbre dimensions

Innovation

• Evaluating joint language-audio embedding models
• Assessing timbre semantics alignment with perception
• Identifying LAION-CLAP as optimal for timbre