RAG-Audio: Retrieval-Augmented Generation for Faithful Brain-to-Audio Reconstruction

2026-08-10

AI Summary

This work addresses the “prior dominance” problem in neural-to-audio reconstruction, where generative models often produce realistic yet content-inaccurate audio under weak neural guidance. To mitigate this, the authors propose a retrieval-guided trajectory initialization mechanism: fMRI signals are first decoded into semantic audio embeddings, which are then used to retrieve matching real audio samples from a reference set. The retrieved sample initializes the sampling trajectory of a frozen diffusion model, while the decoded embedding serves as the generation condition. This approach preserves high-fidelity synthesis while substantially improving alignment between reconstructed audio and the original stimulus. Evaluated on the Brain2Music dataset, the method boosts 10-class stimulus identification accuracy from 0.14–0.18 to 0.40–0.43 and reduces the Fréchet Audio Distance from 13.49 to 1.25, approaching the theoretical upper bound set by retrieval performance.

Abstract

Brain-to-audio reconstruction is limited by \emph{prior domination}: when a pretrained generator is conditioned on a weak neural signal, it produces realistic but stimulus-inaccurate audio. We introduce RAG-Audio, which decodes fMRI into a semantic audio embedding, retrieves a matching real-audio exemplar, and initializes the frozen generator's sampling trajectory from that exemplar while retaining the decoded embedding as conditioning. On Brain2Music, RAG-Audio improves 10-way stimulus identification from $0.14$--$0.18$ for direct generation, near the $0.10$ chance level, to $0.40$--$0.43$, comparable to retrieval. It also reduces Fréchet Audio Distance by roughly an order of magnitude, from $13.49$ to $1.25$ for AudioLDM. RAG-Audio approaches nearest-neighbor retrieval in identification while remaining generative; its higher FAD is expected because retrieval directly replays real audio. An autoregressive negative control, which lacks an initializable latent trajectory, shows no comparable gain, attributing the improvement to trajectory initialization. These results suggest that retrieval-guided initialization can mitigate prior domination in brain-to-audio generation.

Problem

• brain-to-audio reconstruction
• prior domination
• neural decoding
• audio generation
• fMRI

Innovation

• Retrieval-Augmented Generation
• brain-to-audio reconstruction
• prior domination
• trajectory initialization
• fMRI decoding