Joint Feature and Output Distillation for Low-complexity Acoustic Scene Classification

2025-07-25

AI Summary

For low-complexity audio scene classification, this paper proposes a two-stage joint knowledge distillation framework that simultaneously transfers soft label distributions and intermediate-layer feature representations from multiple teachers—PaSST and CP-ResNet—enabling synergistic transfer of semantic knowledge and structural information. Unlike conventional single-stage distillation, our approach enhances the representational capacity of the lightweight student model (CP-Mobile) via soft label averaging and hierarchical feature alignment. Evaluated on the TAU Urban Acoustic Scenes 2022 Mobile dataset, the method achieves a classification accuracy of 59.30% with significantly reduced computational cost. This demonstrates its effectiveness and robustness for acoustic scene recognition on resource-constrained edge devices.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Computer Vision: Scene Analysis & Understanding
• Search and Optimization: Learning to Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

This report presents a dual-level knowledge distillation framework with multi-teacher guidance for low-complexity acoustic scene classification (ASC) in DCASE2025 Task 1. We propose a distillation strategy that jointly transfers both soft logits and intermediate feature representations. Specifically, we pre-trained PaSST and CP-ResNet models as teacher models. Logits from teachers are averaged to generate soft targets, while one CP-ResNet is selected for feature-level distillation. This enables the compact student model (CP-Mobile) to capture both semantic distribution and structural information from teacher guidance. Experiments on the TAU Urban Acoustic Scenes 2022 Mobile dataset (development set) demonstrate that our submitted systems achieve up to 59.30% accuracy.

Problem

• Develop low-complexity acoustic scene classification
• Jointly distill soft logits and feature representations
• Improve accuracy with multi-teacher guidance

Innovation

• Dual-level knowledge distillation framework
• Joint soft logits and feature distillation
• Multi-teacher guidance for compact model