Hearing the Room Through the Shape of the Drum: Modal-Guided Sound Recovery from Multi-Point Surface Vibrations

2026-04-29

AI Summary

This work addresses the significant challenge of recovering sound from everyday solid surfaces that exhibit either strong resonance or weak vibrational responses, where conventional methods struggle to effectively decouple object resonance from the source audio signal. The authors propose a modal-guided acoustic recovery framework that, for the first time, incorporates object vibration modes into multi-point optical vibrometry. By employing a speckle interferometric vibrometry system to capture multi-point, multi-axis vibration signals, they construct a physics-driven acoustic-to-vibrational transfer model and perform multi-signal deconvolution to inversely eliminate resonance effects. This approach overcomes the reliance of existing techniques on highly responsive thin-film materials and demonstrates markedly superior performance across diverse everyday solids—particularly in challenging scenarios—outperforming both single-point speckle vibrometry and other multi-signal fusion methods.

Abstract

Optical vibration sensing enables recovering the scene sound directly from the surface vibration of nearby objects, turning everyday objects into ``visual microphones''. However, most prior methods had focused on capturing the vibrations of specific objects with highly favorable vibration responses. These include objects where the surface vibrations are generated by the object itself (e.g., speaker membrane or guitar body) or objects consisting of a thin membrane which is highly reactive to sound (e.g., a chip bag or the leaf of a plant). In this paper, we tackle sound recovery for a more challenging class of solid objects whose vibration responses are poor or highly resonant. We simultaneously capture vibrations for multiple surface points on the object using a speckle-based vibrometry imaging system. Then, we derive a novel physics-guided vibration formation model that relates the scene sound source to the captured multi-point multi-axis vibrations via the object's vibrational modes. The model is then used to reverse the resonant transfer function of the vibrating object, fusing multiple vibration signals to estimate the original sound source in the scene. We evaluate our approach by recovering sound from a variety of everyday objects, demonstrating that it significantly outperforms traditional single-point speckle vibrometry in challenging scenarios and other signal-processing-based methods for multi-signal fusing.

Problem

• sound recovery
• surface vibrations
• vibrational modes
• solid objects
• resonant transfer

Innovation

• modal-guided sound recovery
• multi-point vibrometry
• speckle-based imaging
• vibrational modes
• resonant transfer function inversion