speech and audio processing

This paper addresses the growing need for automated pattern recognition and modeling in acoustic data analysis. It systematically reviews state-of-the-art machine learning techniques—including deep learning, generative models, and physics-informed neural networks—for acoustic classification, regression, and generation tasks. To bridge the gap between methodology and application, we introduce AcousticsML, an open-source library featuring reproducible Jupyter notebooks and standardized preprocessing pipelines, with unified data interfaces and model evaluation protocols. AcousticsML enables end-to-end acoustic signal analysis and physics-constrained modeling, substantially lowering the barrier to entry for interdisciplinary researchers. By integrating data-driven and mechanism-driven paradigms, the library fosters an open, collaborative research framework for acoustics. It provides scalable, verifiable technical support for real-world applications including environmental noise monitoring, speech enhancement, and structural health diagnostics. (149 words)

To address the challenges of subjective quality assessment and the inefficiency of purely data-driven models in speech/audio coding, this paper proposes a tightly integrated hybrid neural coding framework that synergistically combines model-driven and data-driven paradigms. Methodologically, it introduces a novel multi-level hybrid architecture that deeply couples psychoacoustic-weighted loss, customized time-frequency domain prediction (TF-Codec/MDCTNet), an LPCNet-based backbone, and a neural post-processing module, trained end-to-end via an autoencoder paradigm. The core contribution lies in systematically bridging the performance gap between classical signal modeling and end-to-end deep learning. Experimental results demonstrate that, at ultra-low bitrates of 1.6–3.2 kbps, the proposed method achieves a P.808 MOS gain of ≥0.5 over baselines, yielding subjective audio quality approaching that of wideband codecs, while increasing computational overhead by less than 15%.

To address the longstanding reliance on single-task models and the lack of generalizable representations in computer audition, this paper proposes a systematic framework for constructing Auditory Foundation Models (AFMs). Methodologically, it establishes the core paradigm of AFMs for the first time, integrating unified multi-task modeling, cross-modal (audio–text) aligned representation learning, and instruction-driven human–machine interaction. Technically, the framework encompasses large-scale audio–text contrastive pretraining, multi-task prompt tuning, and self-supervised audio modeling. Experiments demonstrate that the proposed AFM achieves substantial performance gains across 10+ downstream tasks—including automatic speech recognition, sound source separation, and environmental sound classification—while enabling zero-shot transfer and open-domain speech understanding. This work advances computer audition toward generality, multi-task synergy, and natural human–machine interaction.

Existing Audio Large Language Models (AudioLLMs) lack a standardized, comprehensive benchmark for systematically evaluating instruction-following capabilities. Method: We introduce AudioBench—the first multidimensional evaluation benchmark specifically designed for AudioLLMs—covering three core task domains: speech understanding, acoustic scene understanding, and paralinguistic speech understanding. It integrates eight task categories across 26 datasets, including seven newly constructed ones. We formally define an instruction-following evaluation framework, propose a cross-modal instruction assessment protocol, and establish a unified metric system. Contribution/Results: We open-source the evaluation toolkit and a dynamic leaderboard. Comprehensive evaluation of five state-of-the-art AudioLLMs reveals significant capability imbalances across tasks. All data, code, and results are publicly released, establishing a new standard for AudioLLM capability assessment.

To address the lack of unified, standardized evaluation tools for speech, audio, and music signals, this paper introduces the first cross-task, cross-modal, and configurable lightweight evaluation toolkit. The toolkit integrates 65 metrics and 729 configurable variants, supporting multi-source reference evaluation—including waveforms, text transcriptions, and semantic descriptions—across five downstream tasks: audio coding, speech synthesis, speech enhancement, singing voice synthesis, and music generation. Leveraging a Pythonic API, modular metric encapsulation, dependency isolation, and multimodal fusion evaluation techniques, it enables out-of-the-box, end-to-end assessment of both perceptual quality and semantic consistency. Extensive validation on multiple benchmarks confirms its metric diversity and configuration flexibility. The toolkit is open-sourced and has been widely adopted by the research community.

This work addresses the performance gap between convolutional neural networks (CNNs) and pure Transformer architectures in end-to-end raw audio classification—specifically, the inferior accuracy of attention-only models lacking convolutional frontends. To bridge this gap, we propose three key innovations: (1) multi-scale temporal embeddings that jointly capture fine- and coarse-grained temporal structures; (2) a learnable, nonlinear, variable-bandwidth filterbank that replaces handcrafted STFT and fixed preprocessing pipelines; and (3) a CNN-inspired adaptive time-frequency pooling mechanism to enhance local invariance and improve feature compression. Evaluated on FreeSound 50K (200 classes) without pretraining, our model achieves new state-of-the-art performance, significantly outperforming leading CNNs and hybrid architectures in mean average precision (mAP). This is the first demonstration that a purely attention-based model can surpass conventional approaches on raw-audio understanding tasks—establishing both its superiority and practical feasibility.

This work proposes an end-to-end time-domain audio processing framework based on reservoir computing, addressing the limitations of traditional methods that rely on computationally intensive time–frequency transforms such as MFCCs and struggle to balance real-time performance, energy efficiency, and alignment with the human auditory system’s efficacy. By integrating biologically inspired auditory feature extraction with reservoir computing and replacing conventional frequency-domain transformations with lightweight convolutional operations, the proposed approach significantly reduces computational overhead while preserving discriminative feature representation. It eliminates the need for complex preprocessing and enables efficient, low-power real-time speech analysis, making it well-suited for embedded systems and voice-driven applications. This study thus establishes a highly energy-efficient and deployable paradigm for neuromorphic audio processing.

This work addresses stability issues in nnAudio arising from TorchScript incompatibility, edge cases in inverse transforms, and dependency drift in modern PyTorch environments. We refactor the STFT and iSTFT implementations to eliminate dynamic state changes, standardize parameter handling, and explicitly restrict reliable inverse STFT reconstruction to uniformly spaced frequency bins—raising errors for unsupported configurations. Additionally, we ensure that the variable-Q transform (VQT) strictly degenerates to the constant-Q transform (CQT) when γ = 0 and resolve compatibility issues between the cepstral feature pipeline (CFP) and recent SciPy versions. Through differentiable audio transforms, TorchScript-compatible static compilation, and updated dependencies, our implementation achieves robust deployment, validated by comprehensive regression testing, significantly enhancing the reliability and robustness of audio feature extraction for both research and engineering applications.

This work proposes an end-to-end, feature-free audio classification approach based on a parallel deep reservoir computing architecture that operates directly on raw audio waveforms, eliminating the need for explicit feature extraction such as MFCCs. Traditional methods relying on handcrafted features often incur high computational overhead and complex preprocessing pipelines. To evaluate the efficacy of the proposed design, the authors conduct comparative experiments using shallow, serial, and parallel deep reservoir models. Results demonstrate that the parallel architecture achieves significantly superior performance over baseline methods while maintaining low model complexity. The approach enables efficient temporal modeling and hierarchical representation learning, highlighting its scalability and practical potential for audio processing tasks.

This work addresses the error-prone and labor-intensive process of manually rewriting differentiable audio processors for real-time deployment, which often lacks formal verification. To bridge this gap, the authors propose ADAC, a compiler that automatically translates trained differentiable audio models into efficient FAUST code via a framework-agnostic intermediate representation, enabling end-to-end deployment. ADAC supports real-time hot-swapping, incorporates built-in stability verification, and facilitates macro-control parameter design, ensuring that the exported audio plugins match the original model’s impulse response within floating-point precision. Experimental results demonstrate that ADAC successfully converts feedback delay network models into fully functional, stable, and reliable real-time audio plugins, with discrepancies reduced to the level of floating-point noise, thereby establishing a robust pathway from research prototypes to production-grade applications.

Current evaluations of audio understanding are largely confined to automatic speech recognition, failing to capture models’ capabilities in real-world scenarios involving background sounds, noise localization, cross-lingual speech, and non-speech content. To address this gap, this work proposes SCENEBench—the first multidimensional audio understanding benchmark tailored to practical applications such as assistive technologies and industrial noise monitoring. It encompasses four key dimensions: spatial, cross-lingual, environmental, and non-speech understanding. SCENEBench integrates both synthetic and natural audio data, employs a multi-task evaluation protocol with latency measurements, and incorporates an ecological validity verification mechanism. Evaluations of five state-of-the-art large audio-language models reveal significant performance deficiencies across multiple tasks—some even below random chance—highlighting critical shortcomings in understanding “how something is said” and non-speech auditory content, thereby charting a clear direction for future research.