Contextual AD Narration with Interleaved Multimodal Sequence

2024-03-19

arXiv.org

AI Summary

To address the challenge of enabling visually impaired users to comprehend long-form videos, this paper proposes Uni-AD, a framework for generating accurate, role-aligned, and contextually coherent audio descriptions (AD) for film and television. Methodologically: (1) a lightweight video–text feature mapping module is designed to enhance fine-grained cross-modal alignment; (2) a character refinement module and a context-aware decoder are introduced to ensure consistent character naming and effective narrative logic modeling; (3) a contrastive loss is incorporated to optimize multimodal representation learning. Evaluated on multiple AD benchmarks, Uni-AD significantly outperforms state-of-the-art methods, yielding ADs that are more accurate, fluent, and narratively coherent. The implementation code is publicly available.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: AI for Accessibility

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

The Audio Description (AD) task aims to generate descriptions of visual elements for visually impaired individuals to help them access long-form video content, like movies. With video feature, text, character bank and context information as inputs, the generated ADs are able to correspond to the characters by name and provide reasonable, contextual descriptions to help audience understand the storyline of movie. To achieve this goal, we propose to leverage pre-trained foundation models through a simple and unified framework to generate ADs with interleaved multimodal sequence as input, termed as Uni-AD. To enhance the alignment of features across various modalities with finer granularity, we introduce a simple and lightweight module that maps video features into the textual feature space. Moreover, we also propose a character-refinement module to provide more precise information by identifying the main characters who play more significant roles in the video context. With these unique designs, we further incorporate contextual information and a contrastive loss into our architecture to generate smoother and more contextually appropriate ADs. Experiments on multiple AD datasets show that Uni-AD performs well on AD generation, which demonstrates the effectiveness of our approach. Our code is available at: https://github.com/ant-research/UniAD.

Problem

• Generate contextual audio descriptions for visually impaired individuals
• Align multimodal features for accurate character and scene descriptions
• Improve AD quality with character refinement and contrastive loss

Innovation

• Unified framework with interleaved multimodal sequence
• Lightweight module for video-text feature alignment
• Character-refinement module for precise main character identification