ULFine: Unbiased Lightweight Fine-tuning for Foundation-Model-Assisted Long-Tailed Semi-Supervised Learning

2025-05-08

AI Summary

This paper addresses the poor performance of tail classes in long-tailed semi-supervised learning (LTSSL). Existing CLIP-based linear probing or lightweight fine-tuning methods suffer from dual biases—pseudo-label bias and classifier bias—severely limiting tail-class accuracy. To tackle this, we propose ULFine, an unbiased lightweight fine-tuning framework. ULFine introduces a confidence-aware text prototype adaptive fitting mechanism that explicitly decouples and suppresses both biases. Furthermore, it employs a dual-logits complementary fusion strategy to enhance discriminability for tail classes. While incurring extremely low training overhead—over 10× less than state-of-the-art methods—ULFine significantly improves both tail-class accuracy and overall prediction performance. To our knowledge, it is the first lightweight fine-tuning paradigm for LTSSL that systematically mitigates inherent biases in foundation models.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Based on the success of large-scale visual foundation models like CLIP in various downstream tasks, this paper initially attempts to explore their impact on Long-Tailed Semi-Supervised Learning (LTSSL) by employing the foundation model with three strategies: Linear Probing (LP), Lightweight Fine-Tuning (LFT), and Full Fine-Tuning (FFT). Our analysis presents the following insights: i) Compared to LTSSL algorithms trained from scratch, FFT results in a decline in model performance, whereas LP and LFT, although boosting overall model performance, exhibit negligible benefits to tail classes. ii) LP produces numerous false pseudo-labels due to extit{underlearned} training data, while LFT can reduce the number of these false labels but becomes overconfident about them owing to extit{biased fitting} training data. This exacerbates the pseudo-labeled and classifier biases inherent in LTSSL, limiting performance improvement in the tail classes. With these insights, we propose a Unbiased Lightweight Fine-tuning strategy, extbf{ULFine}, which mitigates the overconfidence via confidence-aware adaptive fitting of textual prototypes and counteracts the pseudo-labeled and classifier biases via complementary fusion of dual logits. Extensive experiments demonstrate that ULFine markedly decreases training costs by over ten times and substantially increases prediction accuracies compared to state-of-the-art methods.

Problem

• Explores foundation models' impact on Long-Tailed Semi-Supervised Learning
• Identifies biases in Lightweight Fine-Tuning and Linear Probing methods
• Proposes ULFine to reduce biases and improve tail class performance

Innovation

• Unbiased Lightweight Fine-tuning for LTSSL
• Confidence-aware adaptive textual prototypes
• Complementary fusion of dual logits