A generalized approach to label shift: the Conditional Probability Shift Model

2025-03-04

AI Summary

This paper addresses distribution shift between source and target domains in machine learning caused by conditional probability shift (CPS)—a scenario where class-conditional distributions of certain features change given specific covariates, while others remain invariant. Unlike conventional assumptions of covariate or label shift, CPS captures more fine-grained, feature-specific distributional changes. To tackle this, we propose the first general-purpose CPS modeling framework: the Conditional Probability Shift Model (CPSM). CPSM is classifier-agnostic, requires no prior knowledge of shift type, models class-conditional probabilities via polynomial regression, and estimates parameters using an EM algorithm on unlabeled target data. Experiments on synthetic benchmarks and the MIMIC-III clinical dataset demonstrate that CPSM significantly improves balanced accuracy on target-domain predictions under pure CPS settings, outperforming state-of-the-art domain adaptation and distribution shift correction methods.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

In many practical applications of machine learning, a discrepancy often arises between a source distribution from which labeled training examples are drawn and a target distribution for which only unlabeled data is observed. Traditionally, two main scenarios have been considered to address this issue: covariate shift (CS), where only the marginal distribution of features changes, and label shift (LS), which involves a change in the class variable's prior distribution. However, these frameworks do not encompass all forms of distributional shift. This paper introduces a new setting, Conditional Probability Shift (CPS), which captures the case when the conditional distribution of the class variable given some specific features changes while the distribution of remaining features given the specific features and the class is preserved. For this scenario we present the Conditional Probability Shift Model (CPSM) based on modeling the class variable's conditional probabilities using multinomial regression. Since the class variable is not observed for the target data, the parameters of the multinomial model for its distribution are estimated using the Expectation-Maximization algorithm. The proposed method is generic and can be combined with any probabilistic classifier. The effectiveness of CPSM is demonstrated through experiments on synthetic datasets and a case study using the MIMIC medical database, revealing its superior balanced classification accuracy on the target data compared to existing methods, particularly in situations situations of conditional distribution shift and no apriori distribution shift, which are not detected by LS-based methods.

Problem

• Addresses discrepancy between source and target data distributions.
• Introduces Conditional Probability Shift (CPS) for changing conditional distributions.
• Proposes CPSM model for improved classification accuracy in target data.

Innovation

• Introduces Conditional Probability Shift Model (CPSM)
• Uses multinomial regression for conditional probabilities
• Estimates parameters via Expectation-Maximization algorithm