ESPnet3: Infrastructure for Scalable Speech and Audio Research in the Foundation Model Era

2026-06-19

AI Summary

Existing speech and audio research frameworks often suffer from heavy engineering overhead and poor scalability when handling large-scale data, complex models, and diverse experimental pipelines. To address these limitations, this work proposes a modular, configuration-driven research framework that enables efficient and scalable experimentation through unified Python workflows, a flexible data organization mechanism—including a DataOrganizer abstraction and data sharding—and a lightweight stage-overriding strategy. The framework supports multi-node distributed training with GPU utilization exceeding 80%, reduces each round of OWSM pretraining by 21.1 minutes, and allows integration of new models or datasets with approximately 46 lines of code, substantially enhancing both development efficiency and resource utilization.

Abstract

Recent speech research involves increasingly large datasets, complex models, and diverse experimental workflows. However, existing frameworks require substantial engineering effort to support such experiments. We present ESPnet3, a speech and audio research framework built on a modular system architecture with configuration-driven dataset composition and unified Python-based workflows. ESPnet3 introduces a DataOrganizer abstraction for flexible dataset integration and dataset sharding for memory-efficient large-scale training, while allowing recipe-specific logic through lightweight stage overrides. In OWSM pre-training experiments, ESPnet3 reduces per-epoch training time by \emph{21.1 minutes} compared to ESPnet2 and achieves \emph{>80\% GPU utilization} in multi-node training. Fine-tuning experiments show that new models and datasets can be integrated with around \emph{46 lines of additional code}. ESPnet3 will be publicly released with model checkpoints and training logs.

Problem

• speech research
• large-scale datasets
• complex models
• experimental workflows
• framework scalability

Innovation

• modular architecture
• DataOrganizer
• dataset sharding
• configuration-driven workflow
• scalable speech research