Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation

2025-04-01

AI Summary

Existing audio description (AD) generation methods for film and television editing videos rely heavily on large-scale annotated datasets and model fine-tuning, limiting their applicability in low-resource or zero-shot scenarios. Method: This work proposes a training-free, shot-level AD generation framework leveraging vision-language models (VLMs). It introduces a novel two-stage architecture incorporating shot-level cinematic grammar modeling—covering shot scale, narrative logic, and temporal context—and integrates grammar-guided generation via a knowledge plug-in. Contribution/Results: We propose the Action Score metric and a multi-candidate AD evaluation protocol to enhance action accuracy and narrative coherence. Without any training, our method achieves state-of-the-art zero-shot performance across multiple benchmarks, even surpassing some fine-tuned approaches on key metrics. The Action Score demonstrates strong correlation with human evaluation (ρ > 0.92), validating both effectiveness and practical utility.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Our objective is the automatic generation of Audio Descriptions (ADs) for edited video material, such as movies and TV series. To achieve this, we propose a two-stage framework that leverages"shots"as the fundamental units of video understanding. This includes extending temporal context to neighbouring shots and incorporating film grammar devices, such as shot scales and thread structures, to guide AD generation. Our method is compatible with both open-source and proprietary Visual-Language Models (VLMs), integrating expert knowledge from add-on modules without requiring additional training of the VLMs. We achieve state-of-the-art performance among all prior training-free approaches and even surpass fine-tuned methods on several benchmarks. To evaluate the quality of predicted ADs, we introduce a new evaluation measure -- an action score -- specifically targeted to assessing this important aspect of AD. Additionally, we propose a novel evaluation protocol that treats automatic frameworks as AD generation assistants and asks them to generate multiple candidate ADs for selection.

Problem

• Automatic Audio Description generation for edited videos
• Leveraging shots and film grammar without training VLMs
• Introducing action score and multi-candidate evaluation protocol

Innovation

• Two-stage framework using shots as video units
• Integrates film grammar without VLM training
• Novel action score for AD quality evaluation