Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Models

2026-07-14

AI Summary

This work addresses the challenge that existing text-to-audio models struggle to accurately generate audio containing multiple sound events with precise temporal relationships and lack fine-grained, instruction-level supervision. To overcome this limitation, we introduce, for the first time, an audio-aware large language model as a critic to perform fine-grained verification of both the presence and temporal ordering of target events in generated audio. Leveraging both human and automated feedback, we construct preference pairs and enhance model performance through Direct Preference Optimization. We further propose S3Bench, a new benchmark designed to systematically evaluate multi-event temporal instruction-following capabilities. Experiments demonstrate that our approach significantly improves event completeness, temporal accuracy, and overall instruction adherence across multiple established benchmarks and S3Bench, while preserving high audio generation quality.

Abstract

Recent text-to-audio models generate high-quality audio, but often fail to follow instructions involving multiple sound events and temporal order. This gap arises because existing evaluation and training signals mainly emphasize global similarity or perceptual quality, with limited supervision on instruction-level correctness. We propose an instruction-level framework that uses audio-aware large language models (ALLMs) as fine-grained judges to verify target event presence and temporal relations in generated audio. After validating ALLM judgments on benchmarks and through human verification, we use their feedback to construct preference pairs for direct preference optimization. We further introduce S3Bench, a narrative benchmark for evaluating multi-event temporal instruction following. Experiments show that our method improves event completeness, temporal ordering, and joint instruction-following accuracy across existing benchmarks and S3Bench, while maintaining audio quality.

Problem

• text-to-audio
• instruction following
• temporal order
• multi-event audio
• audio generation

Innovation

• audio-aware LLMs
• instruction following
• fine-grained feedback
• preference optimization
• temporal audio generation