From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation

2026-06-11

AI Summary

This work addresses the high cost, data curation burden, and privacy risks associated with self-supervised audio representation learning that relies on large-scale real-world datasets by proposing AudioPG, a framework enabling efficient pretraining exclusively on procedurally synthesized audio. AudioPG generates waveforms in real time using acoustic primitives and their compositional rules, and trains a Transformer-based masked autoencoder on this synthetic data. The method reveals that physical factors such as fundamental frequency and relative intensity exhibit linearly disentangled structures within orthogonal subspaces of the latent representation, offering both efficiency and interpretability. Evaluated on ESC-50, FSD50K, UrbanSound8K, and Speech Commands V2, the model achieves 90.60%, 0.546 mAP, 88.17%, and 97.03% performance, respectively, with pretraining completing in under 20 minutes on a single GPU.

Abstract

Self-supervised learning advances audio representation for multimedia analysis. However, prevailing data-centric approaches rely on massive real-world corpora, increasing training costs, curation burdens, and privacy barriers. To address this, we present AudioPG, a procedural synthesis framework eliminating real audio recordings during pre-training. AudioPG trains a Transformer-based masked autoencoder on waveforms generated on-the-fly from basic acoustic primitives and composition rules. The encoder transfers effectively to real audio benchmarks, achieving 90.60% accuracy on ESC-50, 0.546 mAP on FSD50K, 88.17% on UrbanSound8K, and 97.03% on Speech Commands V2. Notably, pre-training completes in under 20 minutes on a single GPU. Latent space analysis reveals physical factors, including fundamental frequency and relative intensity, emerge in orthogonal subspaces, making representations linearly decodable. These results establish procedural synthesis as an efficient, interpretable pre-training signal when large-scale corpora are unavailable. Our code is available at: https://github.com/Freyliu0516/audioPG.

Problem

• self-supervised learning
• audio representation
• data scarcity
• privacy barriers
• training cost

Innovation

• procedural generation
• synthetic pre-training
• self-supervised audio learning
• interpretable representation
• masked autoencoder