StrAD: A Streaming Method and Benchmark for Audio Description Generation for Long-form Videos

2026-08-12

AI Summary

This work addresses the limitations of existing audio description generation methods, which struggle with long-form videos and rely on manually annotated timestamps, thereby failing to meet the accessibility needs of visually impaired users at scale. The paper introduces the first streaming framework for audio description generation tailored to long videos, leveraging a sliding window mechanism to enable real-time caption insertion without ground-truth timestamps. The framework supports both fine-tuning (StrAD-FT) and zero-shot prompting with vision-language models (StrAD-Zero). Additionally, the authors construct StrAD, a diverse benchmark of long videos, to standardize full-video-level evaluation. Experiments show that the proposed method achieves a CIDEr score of 36.3 on CMD-AD—outperforming prior work by 10.0—and reaches 51.0 CIDEr on the StrAD benchmark, with a streaming task SODA score of 2.4, substantially exceeding the zero-shot baseline of 1.1.

Abstract

Visual content is the dominant medium of communication, yet without audio descriptions (ADs), it remains inaccessible to blind and low-vision people. ADs narrate context-relevant visual events during natural audio pauses. Manually creating ADs is expensive, limiting coverage to a small fraction of available content. Most existing automatic AD generation methods frame the task as video clip captioning, requiring ground-truth timestamps and additional context cues such as character databases. Current benchmarks reinforce this framing, consisting of short video segments paired with automatic or task-mismatched annotations. We introduce StrAD, a benchmark for long-form AD generation on full-length videos spanning diverse genres such as movies, documentaries, short films, performances, and video games. We reformulate AD generation as streaming dense video captioning. Our approach processes full-length videos with a sliding window, inserting ADs into existing transcripts without ground-truth timestamps, and supports both fine-tuned models and zero-shot prompting of vision-language models. On the segment-level task with given timestamps, our fine-tuned StrAD-FT sets the state of the art on CMD-AD with 36.3 CIDEr (+10.0 over Shot-by-shot), establishes a reference point on StrAD (51.0 CIDEr), and remains competitive on MAD-Eval at 24.9 CIDEr. On the full-video streaming task, StrAD-FT reaches a SODA score of 2.4 against 1.1 for our zero-shot baseline StrAD-Zero, though both exhibit limitations in temporal localization and narrative coherence. While prior work has tackled full-video AD generation in an offline, multi-stage fashion, ours is the first streaming approach, generating ADs on the fly without ground-truth timestamps. StrAD makes progress on full-video AD generation measurable, a prerequisite for scaling accessibility.

Problem

• Audio Description
• Long-form Video
• Streaming Captioning
• Accessibility
• Video Understanding

Innovation

• streaming audio description
• long-form video
• dense video captioning
• zero-shot prompting
• accessibility