Minimax Optimal Two-Stage Algorithm For Moment Estimation Under Covariate Shift

2025-06-29

AI Summary

This paper addresses the classical yet incompletely characterized problem of estimating moments of an unknown function under covariate shift. Methodologically, we propose a two-stage doubly robust truncated estimator: in the first stage, an optimal function estimator is learned from source data; in the second stage, likelihood-ratio weighting—using either known or estimable density ratios—is applied, coupled with truncation to control variance. Our contributions are threefold: (i) we establish, for the first time, the minimax lower bound for this problem; (ii) we prove that the proposed estimator achieves the minimax optimal rate (up to logarithmic factors) when source and target distributions are fully known; and (iii) we demonstrate its strong robustness even when only partial distributional information is available. Extensive synthetic experiments validate both the effectiveness and stability of the estimator.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Intelligent Robots: State Estimation
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements

Abstract

Covariate shift occurs when the distribution of input features differs between the training and testing phases. In covariate shift, estimating an unknown function's moment is a classical problem that remains under-explored, despite its common occurrence in real-world scenarios. In this paper, we investigate the minimax lower bound of the problem when the source and target distributions are known. To achieve the minimax optimal bound (up to a logarithmic factor), we propose a two-stage algorithm. Specifically, it first trains an optimal estimator for the function under the source distribution, and then uses a likelihood ratio reweighting procedure to calibrate the moment estimator. In practice, the source and target distributions are typically unknown, and estimating the likelihood ratio may be unstable. To solve this problem, we propose a truncated version of the estimator that ensures double robustness and provide the corresponding upper bound. Extensive numerical studies on synthetic examples confirm our theoretical findings and further illustrate the effectiveness of our proposed method.

Problem

• Estimating function moments under covariate shift
• Minimax optimal bounds for known distributions
• Robust likelihood ratio estimation for unknown distributions

Innovation

• Two-stage algorithm for moment estimation
• Likelihood ratio reweighting calibration
• Truncated estimator ensuring double robustness