READ More than What You See: Reinforcement Learning for Accurate and Coherent Audio Description Generations

2026-06-21

AI Summary

Existing methods for audio captioning often produce generic, information-poor descriptions lacking narrative coherence due to limitations in training objectives or mismatches in descriptive style. This work addresses these shortcomings by introducing reinforcement learning into audio captioning for the first time, formulating it as a sequence-level optimization problem. The authors propose a context-aware coherence reward mechanism and jointly optimize the generation process using a multi-dimensional reward framework that incorporates reference matching, caption length, and formatting constraints, thereby overcoming the limitations of conventional word-by-word prediction strategies. Evaluated on multiple benchmarks—including MAD-Eval, CMD-AD, and TV-AD—the proposed approach significantly outperforms state-of-the-art methods in both accuracy and narrative coherence.

Abstract

Audio Description aims to generate concise narrations of essential visual content in audio-visual media for blind and low-vision audiences. Existing methods either rely on prompting off-the-shelf multimodal models, which often mismatch AD style, or partially optimize training-based systems with next-token prediction, which under-explores model capacity and biases generation toward generic expressions. We present READ, the first reinforcement-learning (RL) framework for training-based AD generation. READ formulates AD as sequence-level optimization with reference-matching, length, and format rewards, and further introduces a dedicated coherence reward under context-aware supervision to promote narratively coherent descriptions. Experiments on MAD-Eval, CMD-AD, and TV-AD show that READ substantially outperforms prior methods across diverse evaluation metrics. Our results highlight RL as a promising paradigm for accurate and coherent AD generation. Our codes, models, and benchmark results will be publicly available.

Problem

• Audio Description
• Reinforcement Learning
• Coherence
• Accuracy
• Multimodal Generation

Innovation

• reinforcement learning
• audio description
• sequence-level optimization
• coherence reward
• context-aware supervision