FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision

2026-06-11

AI Summary

Existing video-to-audio generation methods struggle to simultaneously achieve multimodal control, frame-level temporal alignment, and fine-grained semantic accuracy. This work proposes a unified conditional generation framework that introduces a novel condition injection mechanism to enable reference audio guidance, designs a multimodal dynamic masking strategy to ensure precise synchronization during training, and integrates an adverb-based data augmentation approach that combines signal processing with large language models to enhance semantic supervision. The proposed method significantly improves the controllability, temporal coherence, and semantic richness of generated audio on the AudioCaps, VGGSound, and Greatest Hits datasets.

Abstract

We present FoleyGenEx, a unified video-to-audio (VTA) framework integrating multi-modal control, frame-level temporal alignment, and fine-grained semantics, enabling synchronized, versatile audio synthesis for diverse tasks. Existing VTA methods either have multi-modal control but weak temporal alignment or strong alignment but lack reference audio conditioning and semantic precision. FoleyGenEx fills this gap via three core innovations: a conditional injection mechanism for audio-controlled VTA and Foley extension, a multi-modal dynamic masking strategy preserving training synchronization, and an adverb-based data augmentation algorithm leveraging signal processing and large language models to enhance textual supervision with nuanced semantics. Experiments on AudioCaps, VGGSound, and Greatest Hits demonstrate its competitive controllable VTA performance against existing methods. Demo samples are available at https://foleygenex.github.io/FoleyGenEx.

Problem

• video-to-audio generation
• temporal alignment
• semantic precision
• multi-modal control
• audio synthesis

Innovation

• video-to-audio generation
• multi-modal control
• temporal alignment
• semantic precision
• conditional audio synthesis