Heterogeneity-Aware Dataset Scheduling for Efficient Audio Large Language Model Training

2026-05-18

AI Summary

This work addresses the challenge of slow convergence in multi-source heterogeneous Audio Question Answering (AudioQA) due to gradient conflicts arising from data heterogeneity during joint training. It introduces the first explicit modeling of inter-dataset heterogeneity by proposing a gradient affinity metric that eliminates the need for empirical transfer evaluation. Building upon this metric, the authors devise a Grouped Sequential Training (GST) strategy that leverages affinity-aware dataset grouping and a progressive scheduling protocol to enhance optimization efficiency while maintaining parallel training stability. The resulting model-agnostic framework achieves 30–40% faster convergence compared to standard parallel training across 14 diverse AudioQA datasets spanning speech, music, and environmental sounds, matching or exceeding the performance of full mixed-data training.

Abstract

Training general-purpose Audio Large Language Models (ALLMs) across diverse datasets is essential for holistic audio understanding, yet it faces significant challenges due to dataset heterogeneity, which often leads to conflicting gradients and slow convergence. Despite its impact, how to explicitly manage this heterogeneity during training remains underexplored, with current practices relying primarily on uniform mixture. In this work, we analyze multi-dataset AudioQA training from a convergence perspective and propose Grouped Sequential Training (GST). GST strategically organizes datasets into affinity-aware groups and introduces them via a progressive scheduling protocol, effectively balancing the stability of parallel training with the efficiency of sequential optimization. To ensure scalability, we develop gradient-based affinity metrics that capture inter-dataset relationships without the prohibitive cost of empirical transferability estimation. Extensive evaluations on 14 AudioQA datasets spanning speech, music, and environmental sounds demonstrate that GST achieves 30--40\% faster convergence than standard parallel training while maintaining or even surpassing the performance of mix-all training. Our results provide both theoretical insights and a practical, model-agnostic framework for efficient large-scale ALLM optimization.

Problem

• dataset heterogeneity
• Audio Large Language Models
• convergence
• multi-dataset training
• gradient conflict

Innovation

• Grouped Sequential Training
• dataset heterogeneity
• gradient-based affinity
• Audio Large Language Models
• convergence acceleration