Multi-Stage Music Source Restoration with BandSplit-RoFormer Separation and HiFi++ GAN

2026-03-04

AI Summary

This work proposes a two-stage music source restoration method to address the nonlinear mixing artifacts present in full-mix audio, which arise from production effects and distribution-induced distortions. The approach first leverages a BandSplit-RoFormer architecture, enhanced with a three-phase curriculum learning strategy, LoRA-based fine-tuning, and head expansion, to efficiently separate eight instrument stems. Subsequently, it employs both universal and instrument-specific HiFi++ GAN-based waveform restorers to jointly optimize audio quality and fidelity. By moving beyond the conventional linear mixture assumption, the proposed method achieves significant improvements in stem separation quality and audio reconstruction performance on the ICASSP MSR Challenge 2025 benchmark.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Mixed Discrete/Continuous Search
• Planning, Routing, and Scheduling: Mixed Discrete/Continuous Planning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Music Source Restoration (MSR) targets recovery of original, unprocessed instrument stems from fully mixed and mastered audio, where production effects and distribution artifacts violate common linear-mixture assumptions. This technical report presents the CP-JKU team's system for the MSR ICASSP Challenge 2025. Our approach decomposes MSR into separation and restoration. First, a single BandSplit-RoFormer separator predicts eight stems plus an auxiliary other stem, and is trained with a three-stage curriculum that progresses from 4-stem warm-start fine-tuning (with LoRA) to 8-stem extension via head expansion. Second, we apply a HiFi++ GAN waveform restorer trained as a generalist and then specialized into eight instrument-specific experts.

Problem

• Music Source Restoration
• audio restoration
• stem separation
• mastered audio
• nonlinear mixture

Innovation

• Music Source Restoration
• BandSplit-RoFormer
• HiFi++ GAN
• multi-stage curriculum learning
• instrument-specific restoration