Problem
Research questions and friction points this paper is trying to address.
audio-to-image generation
cross-modal alignment
high-fidelity images
multimodal dataset
expressive synthesis
Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Framework
🤖 AI Summary
This work addresses the limitations of existing audio-to-image generation methods, which struggle to effectively fine-tune advanced text-to-image models due to the scarcity of high-quality, cross-modally aligned datasets. To overcome this challenge, the authors introduce A2I-Set, the first large-scale trimodal-aligned dataset comprising 323,000 triples of audio clips, images, and fine-grained textual descriptions, alongside a dedicated generative framework named AudioCanvas. AudioCanvas is fine-tuned on A2I-Set to enable audio-driven image synthesis. The study further proposes a human-supervised, mixed-source test set to rigorously evaluate cross-modal consistency. Experimental results demonstrate that AudioCanvas significantly outperforms current approaches in both visual fidelity and semantic alignment between audio and generated images, thereby validating the effectiveness of the proposed dataset and model architecture.
📝 Abstract
As an important subfield of cross-modal generation, synthesizing static visual content in the form of images from audio, namely audio-to-image (A2I) generation, has attracted increasing research attention in recent years. Nevertheless, despite the remarkable visual quality of modern text-to-image (T2I) models, the performance of A2I remains fundamentally limited by traditional datasets, which often lack both high-fidelity images and precise cross-modal alignment. As a result, existing methods still struggle to achieve high-quality audio-to-image generation through finetuning strong T2I models, thereby constraining practical applications in this area. Motivated by this gap, we introduce A2I-Set, a unified, high-quality tri-modal dataset consisting of 323K paired audio, images, and detailed text captions, specifically designed for audio-visual research, including audio-conditioned image generation. Besides, we developed a new mixed-source test set for the A2I task through human supervision. We further propose an A2I model, AudioCanvas, fine-tuned on our A2I-Set. Experiments show that AudioCanvas achieves more visually expressive as well as cross-modal alignment results that generally outperforming existing approaches. Our dataset and source code are available at https://github.com/gdx012/A2I-Generation.
Innovation
Methods, ideas, or system contributions that make the work stand out.
audio-to-image generation
multimodal dataset
cross-modal alignment
AudioCanvas
high-fidelity synthesis
🔎 Similar Papers
No similar papers found.
