Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections

2026-08-11

AI Summary

This work addresses the limitations of existing video-to-music generation research, which often relies on non-reproducible and copyright-infringing web-crawled data. To overcome this, we introduce OSSL-v2, a large-scale, self-hostable, and copyright-compliant video–music paired dataset constructed exclusively from public-domain films. We further present the first systematic exploration of leveraging film dialogue as a temporal modulation signal for music generation, employing a time-aware cross-modal attention mechanism to fuse per-frame dialogue audio features into the generation process. Experimental results demonstrate that our approach significantly outperforms current baselines on both public-domain and commercial movie test sets, validating the efficacy of dialogue-aware conditioning in enhancing the quality and alignment of generated soundtracks.

Abstract

Video-to-music generation has drawn growing interest for its role in conveying the emotion of visual media, including film. Progress in the field, however, is hampered by a reproducibility gap: models are often trained on crawled corpora referenced through YouTube URLs that may be deleted, with the underlying data often difficult and time-consuming to retrieve. To address this, we introduce the Open Screen Soundtrack Library version 2 (OSSL-v2), a self-hosted corpus of 34,343 video clips totaling 246.4 hours, sourced from public-domain films. Unlike crawled corpora, OSSL-v2 is reproducible (i.e., not subject to link rot) and copyright-conscious, yet still large enough to train functional video-to-music models. We then use this film-domain corpus to study dialogue as a conditioning signal for video-to-music generation, motivated by the close temporal coupling between film music and on-screen speech. Specifically, we augment existing models' video cross-attention with a time axis and modulate it frame-by-frame with the dialogue track. Evaluated on both public-domain and commercial films, our approach shows improvement over the state-of-the-art baselines. The dataset is available at https://huggingface.co/datasets/McAuley-Lab/OSSL-v2.

Problem

• video-to-music generation
• reproducibility gap
• link rot
• public-domain films
• dialogue-aware

Innovation

• video-to-music generation
• public domain films
• dialogue-aware conditioning
• reproducible dataset
• cross-attention modulation