Investigating the Robustness of Subtask Distillation under Spurious Correlation

2026-01-31

AI Summary

This study systematically evaluates the robustness of various knowledge distillation methods under limited or non-representative data exhibiting spurious correlations, with a particular focus on SubDistill. To the best of our knowledge, this is the first comprehensive assessment of how such methods perform as the strength of spurious correlations varies. Experimental results demonstrate that the performance gap between advanced distillation approaches and conventional baselines widens significantly with increasing spurious correlation strength. Notably, SubDistill maintains robust performance even under strong spurious interference, whereas several baseline methods degrade to near-random accuracy. These findings highlight critical challenges in applying knowledge distillation to real-world scenarios where training data may be biased or unrepresentative, and provide empirical insights for designing more robust distillation strategies.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Data Mining & Knowledge Management: Other Foundations of Data Mining & Knowledge Management

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Subtask distillation is an emerging paradigm in which compact, specialized models are extracted from large, general-purpose'foundation models'for deployment in environments with limited resources or in standalone computer systems. Although distillation uses a teacher model, it still relies on a dataset that is often limited in size and may lack representativeness or exhibit spurious correlations. In this paper, we evaluate established distillation methods, as well as the recent SubDistill method, when using data with spurious correlations for distillation. As the strength of the correlations increases, we observe a widening gap between advanced methods, such as SubDistill, which remain fairly robust, and some baseline methods, which degrade to near-random performance. Overall, our study underscores the challenges of knowledge distillation when applied to imperfect, real-world datasets, particularly those with spurious correlations.

Problem

• subtask distillation
• spurious correlation
• knowledge distillation
• robustness
• foundation models

Innovation

• subtask distillation
• spurious correlation
• robustness
• knowledge distillation
• foundation models