COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representations

2024-04-25

arXiv.org

AI Summary

This paper addresses the lack of interpretable, reference-free quality assessment methods for generative music accompaniment. We propose an audio contrastive learning framework explicitly designed to evaluate harmonic and rhythmic consistency between accompaniment and melody. Methodologically, we introduce voice-level consistency modeling into contrastive learning for the first time, leveraging multi-voice features extracted via harmonic-percussive source separation (HPS) to construct audio representations that capture fine-grained harmonic compatibility and rhythmic alignment. Our contributions are threefold: (1) the first voice-level contrastive learning paradigm enabling reference-free, interpretable evaluation; (2) empirical validation on four public multi-instrument datasets—including MUSDB18-HQ—demonstrating significantly improved correlation between automated scores and human perceptual judgments; and (3) open-sourcing of both model and code to advance standardization of objective evaluation for generative accompaniment.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Generation
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present COCOLA (Coherence-Oriented Contrastive Learning for Audio), a contrastive learning method for musical audio representations that captures the harmonic and rhythmic coherence between samples. Our method operates at the level of the stems composing music tracks and can input features obtained via Harmonic-Percussive Separation (HPS). COCOLA allows the objective evaluation of generative models for music accompaniment generation, which are difficult to benchmark with established metrics. In this regard, we evaluate recent music accompaniment generation models, demonstrating the effectiveness of the proposed method. We release the model checkpoints trained on public datasets containing separate stems (MUSDB18-HQ, MoisesDB, Slakh2100, and CocoChorales).

Problem

• Music Understanding
• Harmony and Rhythm Consistency
• Computer-generated Music Evaluation

Innovation

• Contrastive Learning
• Music Harmony
• Rhythm Consistency