MMAudioReverbs: Video-Guided Acoustic Modeling for Dereverberation and Room Impulse Response Estimation

2026-05-01

AI Summary

Existing video-to-audio (V2A) generation models lack explicit modeling of physical acoustic effects such as reverberation and room impulse responses (RIRs), limiting their controllability. This work proposes a unified framework that leverages visual cues to guide dereverberation and RIR estimation by fine-tuning a pretrained MMAudio model on a small-scale dataset, without modifying its network architecture. To the best of our knowledge, this is the first approach to treat a foundational V2A model as prior knowledge for physical acoustic analysis. Experimental results demonstrate that audiovisual cues exhibit complementary strengths across diverse room acoustic tasks, thereby validating the effectiveness and potential of pretrained V2A models in physical acoustic modeling.

Abstract

Although recent video-to-audio (V2A) models excelled at synthesizing semantically plausible sounds from visual inputs, they do not explicitly model room-acoustic effects such as reverberation or room impulse responses (RIRs), and thus offer limited controllability over these effects. However, we hypothesize that such V2A models implicitly have semantic knowledge of the relationship between spatial audio and the corresponding vision cues. In this paper, we revisit a V2A model for the sake of the above, and propose the way to utilize the pretrained model as prior for physically grounded room-acoustic processing. Based on one of the state-of-the-art V2A models, MMAudio, we propose MMAudioReverbs that is a unified framework dealing with i) dereverberation and ii) room impulse response (RIR) estimation without network architectural modification, and fine-tuned on a small dataset. Experimental results showed that audio and visual cues respectively have advantage depending on the type of physical room acoustics. It implies that foundation V2A models can be used for physically grounded room-acoustic analysis.

Problem

• video-to-audio
• reverberation
• room impulse response
• acoustic modeling
• dereverberation

Innovation

• video-to-audio
• room impulse response
• dereverberation
• acoustic modeling
• foundation model