Improving realistic semi-supervised learning with doubly robust estimation

2025-02-01

AI Summary

In semi-supervised learning (SSL) under long-tailed label distributions, pseudo-labeling suffers from severe bias due to unknown class priors over unlabeled data. To address this, this work introduces Doubly Robust Estimation (DRE) into SSL for the first time, enabling explicit, high-accuracy estimation of the unlabeled data’s class prior distribution, with rigorous theoretical guarantees. Our method jointly integrates class-distribution estimation, pseudo-label refinement, and long-tail modeling into a unified end-to-end SSL training framework. Extensive experiments on multiple long-tailed SSL benchmarks demonstrate consistent and substantial improvements over state-of-the-art methods—including FixMatch and FlexMatch—particularly for tail classes, where accuracy increases by 5.2–9.8 percentage points on average. These results validate both the effectiveness and generalizability of our approach in mitigating pseudo-label bias under imbalanced data regimes.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Intelligent Robots: State Estimation
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

A major challenge in Semi-Supervised Learning (SSL) is the limited information available about the class distribution in the unlabeled data. In many real-world applications this arises from the prevalence of long-tailed distributions, where the standard pseudo-label approach to SSL is biased towards the labeled class distribution and thus performs poorly on unlabeled data. Existing methods typically assume that the unlabeled class distribution is either known a priori, which is unrealistic in most situations, or estimate it on-the-fly using the pseudo-labels themselves. We propose to explicitly estimate the unlabeled class distribution, which is a finite-dimensional parameter, emph{as an initial step}, using a doubly robust estimator with a strong theoretical guarantee; this estimate can then be integrated into existing methods to pseudo-label the unlabeled data during training more accurately. Experimental results demonstrate that incorporating our techniques into common pseudo-labeling approaches improves their performance.

Problem

• Semi-supervised Learning
• Imbalanced Data
• Pseudo Labeling

Innovation

• Semi-Supervised Learning
• Pseudo Labeling
• Long-tailed Distribution