BackgroundMellow: A Multi-Modal Cohesive Framework for Narrative-Driven Rich Cinematic Soundscape Generation

2026-07-13

AI Summary

Existing approaches struggle to generate immersive, synchronized soundscapes from long-form narrative texts that simultaneously preserve narrative coherence, temporal alignment, and cinematic emotional depth. This work proposes a teacher-student agent framework that operates without ground-truth audio labels, formulating story-to-audio generation as a multi-track audio orchestration problem. An NLP-driven parameter prediction module parses the input text to estimate onset time, duration, and loudness for each audio track. Semantic-consistent sound effects and musical scores are then synthesized using a Tango2-based latent diffusion model for ambient sounds and a film-score nearest-neighbor retrieval system, respectively. These components are automatically mixed into a unified soundscape. Experiments on a YouTube movie trailer dataset demonstrate that the proposed method significantly outperforms baseline approaches in temporal synchronization, sound effect coverage, and spectral richness.

Abstract

Generating immersive, synchronized and cinematic audio for long-form textual narratives remains a significant challenge in multi-modal AI. While current Text-to-Audio (TTA) frameworks successfully synthesize isolated sound effects, they struggle with narrative cohesion, temporal alignment, and cinematic emotional depth. We present BackgroundMellow, a framework that treats story-to-audio generation as a precise orchestration and signal processing problem. This framework is enabled without ground-truth through a master-specialist agent architecture that decomposes text into precise and multi-layered audio cues, generates each category of sounds with suitable specialist model, and superimposes the soundscapes to create a unified and aligned audio segment. Our pipeline is built over Tango2 latent diffusion model for environmental synthesis alongside a novel Cinematic BGM Retriever mined from professional soundtracks. To automate the sound mixing process, we use an NLP based module that predicts precise audio parameters, like start time, duration, and relative loudness, based on the narrative timeline. We further empirically evaluate and show the efficacy of the proposed framework leveraging nearest-neighbor retrieval against a curated dataset of YouTube cinematic trailers to measure temporal synchronization, coverage, and spectral richness.

Problem

• narrative-driven audio generation
• multi-modal AI
• cinematic soundscape
• temporal alignment
• narrative cohesion

Innovation

• multi-modal audio generation
• narrative-driven soundscapes
• master-specialist architecture
• cinematic BGM retrieval
• automated audio mixing