From Masking to Merging: Rethinking SpecAugment for Efficient Audio Spectrogram Transformer

2026-09-06

AI Summary

本文提出了一种通过在补丁级别应用SpecAugment并合并掩码补丁来加速音频频谱图Transformer训练的方法，提高了处理速度且性能损失很小。

Abstract

This paper proposes SpecAugment-Patch Merging, a simple yet effective method to accelerate Audio Spectrogram Transformer (AST) training. We first apply SpecAugment to mask input spectrograms at the patch level, and after positional embeddings are added, the method selects r pairs of masked patches and merges them, reducing the number of tokens processed by the Transformer. Increasing the number of merged pairs r from 0 to 100 keeps mAP on AudioSet nearly unchanged (34.07 to 34.08) while throughput increases from 43.3 to 49.3 samples/sec, which is a relatively 13.9% improvement. Similar patterns appear on ESC-50 and Speech Commands V2, where throughput steadily improves with only minor accuracy changes, demonstrating that this merging approach provides faster training with minimal performance loss.

Problem

• SpecAugment
• Audio Spectrogram Transformer
• training efficiency

Innovation

• SpecAugment
• Patch Merging
• Audio Spectrogram Transformer
• Efficient Training