Taste-aware music retrieval from audio embeddings

2026-07-03

AI Summary

This study addresses the prediction of five basic taste perceptions associated with music directly from raw audio signals and their application to content-based retrieval. The authors formulate the auditory–gustatory mapping as a multi-task regression problem, leveraging a frozen audio encoder from the HEAR framework, a gated late fusion strategy, and a shared regression head. Validity is confirmed through ridge regression probing and frequency-band ablation analyses. This work establishes the first operational benchmark for taste-aware music retrieval grounded in psychophysically validated cross-modal correspondences. The best-performing system achieves a macro RMSE of 0.134—half the magnitude of human inter-rater variability—and substantially outperforms the previous state-of-the-art (RMSE: 0.219). Furthermore, retrieval performance in the taste-perception space significantly surpasses that of CLAP-based text baselines.

Abstract

Crossmodal correspondences between sound and taste are well established in psychology and neuroscience, but largely absent from content-based multimedia retrieval. We formalise taste-from-audio prediction as a content-based music information retrieval benchmark over a perceptually validated multi-source corpus, comparing ten frozen audio encoders from the four HEAR families under a shared multi-task regression head, with gated late-fusion as a configurable variant. In order to assess the effectiveness of the models, we compute absolute error and rank correlation. The strongest systems predict the five tastes within a macro RMSE of 0.134; on held-out real music their error is less than half a single rater's deviation from the consensus (RMSE 0.13 vs. 0.28), so the model tracks the group consensus more closely than an average human rater, and well below the previous state of the art baseline (0.219). On absolute error the encoders are statistically flat, with a single VGGish matching the best fusion, but gated late-fusion's advantage is confined to rank correlation (macro Pearson r 0.724 vs. 0.666). Operationalised as a content-based retrieval index, the predicted taste space ranks a 309-item pool far more faithfully than a CLAP-text baseline, which sits at chance; ridge probes and an audio-bandstop knockout read the strongest representations against documented sound-taste correspondences.

Problem

• taste-aware retrieval
• crossmodal correspondence
• music information retrieval
• audio embeddings
• content-based retrieval

Innovation

• crossmodal retrieval
• taste prediction
• audio embeddings
• gated late-fusion
• music information retrieval