Improving Text-to-Music Generation with Human Preference Rewards

2026-06-19

AI Summary

This work aims to enhance the audio quality and alignment with human preferences in text-to-music generation. Building upon the 120M-parameter FluxAudio-S model, it introduces a novel approach that integrates the TuneJury human preference reward both as a conditioning signal during training and as a selection criterion during inference. This is combined with a multi-stage optimization framework comprising CRPO-based preference fine-tuning, expert iteration, and joint classifier-free guidance (CFG). Evaluation on 100 Song Describer prompts demonstrates that reward conditioning effectively establishes a functional axis for generation control, with expert iteration yielding the most substantial gains. Collectively, these strategies significantly improve both semantic consistency between generated music and textual descriptions and overall human preference scores.

Abstract

We describe our entry to the efficiency track of the Academic Text-to-Music (ATTM) Grand Challenge at ICME 2026. Beyond the challenge protocol's FAD-CLAP and CLAP score, we add a learned human-preference reward from TuneJury, a twin pairwise ranker trained over open music-preference datasets. The reward serves both as a training-time conditioning signal and as a sample-selection criterion. The pipeline combines five engineering decisions on a 120M-parameter FluxAudio-S backbone, four at training time and one at inference: (i) training-time reward conditioning that doubles as an inference-time CFG axis, (ii) a sweep over five score-conditioning architectures, where training and inference use different variants, (iii) expert iteration on the top decile, (iv) a short preference-tuning pass (CRPO) for audio-text alignment, and (v) inference post-processing via joint CFG, source separation, and loudness normalization. Per-stage decomposition on 100 Song Describer prompts shows training-time reward conditioning as a functional conditioning axis, expert iteration as the dominant contributor, the preference-tuning pass adding only noise-level gain, and the inference-time score scalar already saturated by the end of the chain.

Problem

• text-to-music generation
• human preference
• music generation
• preference reward
• audio-text alignment

Innovation

• human preference reward
• text-to-music generation
• reward conditioning
• expert iteration
• classifier-free guidance