Semi-Supervised Regression with Heteroscedastic Pseudo-Labels

2025-10-16

AI Summary

In semi-supervised regression, pseudo-labels are often corrupted by heteroscedastic noise, leading to unreliable confidence estimation, error accumulation, and overfitting. To address this, we propose an uncertainty-aware pseudo-labeling framework—the first to systematically integrate uncertainty modeling into pseudo-label generation for semi-supervised regression. Our method employs a bilevel optimization scheme that jointly learns the regression model and a heteroscedastic noise estimator: the upper level calibrates pseudo-label confidence via uncertainty quantification, while the lower level optimizes the prediction model using confidence-weighted empirical risk minimization. This enables adaptive noise calibration and significantly improves generalization. Extensive experiments on multiple benchmark datasets demonstrate that our approach surpasses existing state-of-the-art methods in both robustness and prediction accuracy. The implementation is publicly available.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Pseudo-labeling is a commonly used paradigm in semi-supervised learning, yet its application to semi-supervised regression (SSR) remains relatively under-explored. Unlike classification, where pseudo-labels are discrete and confidence-based filtering is effective, SSR involves continuous outputs with heteroscedastic noise, making it challenging to assess pseudo-label reliability. As a result, naive pseudo-labeling can lead to error accumulation and overfitting to incorrect labels. To address this, we propose an uncertainty-aware pseudo-labeling framework that dynamically adjusts pseudo-label influence from a bi-level optimization perspective. By jointly minimizing empirical risk over all data and optimizing uncertainty estimates to enhance generalization on labeled data, our method effectively mitigates the impact of unreliable pseudo-labels. We provide theoretical insights and extensive experiments to validate our approach across various benchmark SSR datasets, and the results demonstrate superior robustness and performance compared to existing methods. Our code is available at https://github.com/sxq/Heteroscedastic-Pseudo-Labels.

Problem

• Addressing heteroscedastic noise in semi-supervised regression pseudo-labels
• Mitigating error accumulation from unreliable continuous pseudo-labels
• Developing uncertainty-aware framework for robust pseudo-label influence adjustment

Innovation

• Uncertainty-aware pseudo-labeling framework for regression
• Dynamically adjusts pseudo-label influence via bi-level optimization
• Jointly minimizes empirical risk and optimizes uncertainty estimates