REFRAMED: Towards Realistic Audio Description Generation for Movies

2026-08-10

AI Summary

This work addresses a critical limitation in existing audio captioning approaches, which typically generate descriptions under predefined content and timing constraints, thereby overlooking the real-world challenge of jointly deciding *what* to describe and *when* to describe it, compounded by a scarcity of high-quality aligned data. To bridge this gap, the study introduces a more realistic task formulation that requires models to simultaneously determine descriptive content and its temporal placement. The authors construct REFRAMED, a large-scale movie dataset featuring professionally authored audio descriptions meticulously aligned with subtitles and screenplays. This work pioneers modeling audio captioning as a joint decision-making task and releases a multi-reference dataset alongside a dialogue-gap-aware evaluation protocol that supports narrative context modeling. Experiments reveal that while current systems outperform baselines, they still fall significantly short of human experts, establishing REFRAMED as a robust benchmark and foundational resource for future research.

Abstract

Audio Description (AD) is a verbal narration of key visual content in videos, enabling access for visually impaired audiences. Unlike standard video captioning, AD is a structured editorial task: descriptions must be inserted into gaps in dialogue and must convey only what is needed to understand the narrative being told. However, existing approaches formulate AD generation in an artificial setting where both the content and timing of descriptions are pre-specified, reducing the task to clip-level captioning. They further rely on noisy transcription and alignment pipelines, and lack the rich parallel data required for modeling narrative context. We introduce a new formulation of AD generation in which models must jointly decide what to describe and when to do it. To support this, we present REFRAMED, a high-quality dataset of 2,023 videos that span 3,302 scenes from 206 movies, with professional AD transcripts (both American and British versions), professional subtitles and aligned screenplays. We also provide a manually curated challenge set that pairs full movies with multiple AD references, together with evaluation protocols that leverage dialogue gaps and multi-reference comparisons. Experiments with state-of-the-art AD systems and multimodal LLMs show that they outperform trivial baselines but fall far short of expert human performance. Our dataset and benchmark establish a new foundation for research on video understanding.

Problem

• Audio Description
• video understanding
• narrative context
• accessibility
• multimodal generation

Innovation

• Audio Description
• Multimodal Understanding
• Narrative Context
• Temporal Planning
• REFRAMED Dataset