🤖 AI Summary
This work addresses the performance gap between convolutional neural networks (CNNs) and pure Transformer architectures in end-to-end raw audio classification—specifically, the inferior accuracy of attention-only models lacking convolutional frontends. To bridge this gap, we propose three key innovations: (1) multi-scale temporal embeddings that jointly capture fine- and coarse-grained temporal structures; (2) a learnable, nonlinear, variable-bandwidth filterbank that replaces handcrafted STFT and fixed preprocessing pipelines; and (3) a CNN-inspired adaptive time-frequency pooling mechanism to enhance local invariance and improve feature compression. Evaluated on FreeSound 50K (200 classes) without pretraining, our model achieves new state-of-the-art performance, significantly outperforming leading CNNs and hybrid architectures in mean average precision (mAP). This is the first demonstration that a purely attention-based model can surpass conventional approaches on raw-audio understanding tasks—establishing both its superiority and practical feasibility.
📝 Abstract
Over the past two decades, CNN architectures have produced compelling models of sound perception and cognition, learning hierarchical organizations of features. Analogous to successes in computer vision, audio feature classification can be optimized for a particular task of interest, over a wide variety of datasets and labels. In fact similar architectures designed for image understanding have proven effective for acoustic scene analysis. Here we propose applying Transformer based architectures without convolutional layers to raw audio signals. On a standard dataset of Free Sound 50K,comprising of 200 categories, our model outperforms convolutional models to produce state of the art results. This is significant as unlike in natural language processing and computer vision, we do not perform unsupervised pre-training for outperforming convolutional architectures. On the same training set, with respect mean aver-age precision benchmarks, we show a significant improvement. We further improve the performance of Transformer architectures by using techniques such as pooling inspired from convolutional net-work designed in the past few years. In addition, we also show how multi-rate signal processing ideas inspired from wavelets, can be applied to the Transformer embeddings to improve the results. We also show how our models learns a non-linear non constant band-width filter-bank, which shows an adaptable time frequency front end representation for the task of audio understanding, different from other tasks e.g. pitch estimation.