Compressing Model with Few Class-Imbalance Samples: An Out-of-Distribution Expedition

2025-02-09

AI Summary

Model compression under few-shot learning with severe class imbalance suffers significant performance degradation. Method: We propose OE-FSMC, an adaptive compression framework that leverages readily available out-of-distribution (OOD) data. It dynamically rebalances the training distribution during knowledge distillation and fine-tuning—first systematically revealing how class imbalance impairs few-shot compression. Innovatively, OE-FSMC introduces an OOD augmentation mechanism and jointly optimizes a distillation loss with L2 regularization and feature-level consistency constraints to simultaneously correct distribution bias and suppress OOD overfitting. Results: Evaluated on multiple benchmark datasets, OE-FSMC substantially mitigates accuracy degradation induced by class imbalance and serves as a plug-and-play module that consistently enhances the performance of existing few-shot compression methods.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Data Mining & Knowledge Management: Data Compression
• Search and Optimization: Distributed Search

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

In recent years, as a compromise between privacy and performance, few-sample model compression has been widely adopted to deal with limited data resulting from privacy and security concerns. However, when the number of available samples is extremely limited, class imbalance becomes a common and tricky problem. Achieving an equal number of samples across all classes is often costly and impractical in real-world applications, and previous studies on few-sample model compression have mostly ignored this significant issue. Our experiments comprehensively demonstrate that class imbalance negatively affects the overall performance of few-sample model compression methods. To address this problem, we propose a novel and adaptive framework named OOD-Enhanced Few-Sample Model Compression (OE-FSMC). This framework integrates easily accessible out-of-distribution (OOD) data into both the compression and fine-tuning processes, effectively rebalancing the training distribution. We also incorporate a joint distillation loss and a regularization term to reduce the risk of the model overfitting to the OOD data. Extensive experiments on multiple benchmark datasets show that our framework can be seamlessly incorporated into existing few-sample model compression methods, effectively mitigating the accuracy degradation caused by class imbalance.

Problem

• Class imbalance in few-sample model compression
• Integration of out-of-distribution data
• Mitigating accuracy degradation in compressed models

Innovation

• Integrates out-of-distribution data
• Uses joint distillation loss
• Incorporates regularization term