Transductive Model Selection under Prior Probability Shift

2025-07-30

AI Summary

Under prior probability shift in transductive learning, conventional cross-validation—relying on labeled training data—fails to ensure effective model selection on the unlabeled target domain. Method: We propose a target-distribution-aware transductive hyperparameter optimization framework that constructs a surrogate performance evaluation criterion directly on unlabeled test samples, without requiring ground-truth labels. It enables adaptive model selection via consistency regularization and pseudo-label stability modeling. Contribution/Results: This is the first work to systematically integrate model selection into transductive classification under prior shift. Experiments across multiple benchmark datasets demonstrate that our method significantly outperforms standard cross-validation, yielding average accuracy improvements of 2.3–5.1 percentage points, while exhibiting superior robustness to varying degrees of prior shift.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Transductive learning is a supervised machine learning task in which, unlike in traditional inductive learning, the unlabelled data that require labelling are a finite set and are available at training time. Similarly to inductive learning contexts, transductive learning contexts may be affected by dataset shift, i.e., may be such that the IID assumption does not hold. We here propose a method, tailored to transductive classification contexts, for performing model selection (i.e., hyperparameter optimisation) when the data exhibit prior probability shift, an important type of dataset shift typical of anti-causal learning problems. In our proposed method the hyperparameters can be optimised directly on the unlabelled data to which the trained classifier must be applied; this is unlike traditional model selection methods, that are based on performing cross-validation on the labelled training data. We provide experimental results that show the benefits brought about by our method.

Problem

• Optimizing model selection under prior probability shift
• Addressing dataset shift in transductive classification contexts
• Improving hyperparameter optimization using unlabelled target data

Innovation

• Transductive model selection for prior probability shift
• Hyperparameter optimization on unlabelled data
• Anti-causal learning problem solution