Deep Regression for Repeated Measurements under Covariate Shift

2026-05-24

AI Summary

This work proposes a nonparametric regression method based on transfer learning for covariate shift settings where the response variable in the target domain is either unobservable or prohibitively expensive to acquire. Leveraging labeled data from the source domain and replicate measurements from the target domain, the approach jointly estimates the density ratio and the target regression function using ReLU feedforward neural networks. A novel approximation theory is introduced, ensuring that the network parameters depend only polynomially on the input dimension, thereby effectively mitigating the curse of dimensionality. Theoretical analysis establishes non-asymptotic error bounds that achieve minimax optimal convergence rates. Numerical experiments and real-data analyses further demonstrate the method’s superior finite-sample performance.

Abstract

This paper studies nonparametric regression with repeated measurements when the response in the target domain is unobservable or costly to collect. We adopt a transfer learning framework that leverages a source domain with observable responses under covariate shift. The target regression function is estimated by correcting the distribution shift via the density ratio. We consider both known and unknown density ratio scenarios, which reflect different data available for nonparametric regression estimation. In both cases, we further address two settings: the uniformly bounded density ratio and the unbounded case with finite moment conditions. Under the unknown density ratio scenario, both the density ratio and the target regression function are estimated using rectified linear unit (ReLU) feedforward neural networks (FNNs), whereas under the known density ratio scenario, only the target regression function is estimated by ReLU FNNs. Theoretically, we establish non-asymptotic error bounds for the proposed estimators and prove that they achieve the minimax optimal convergence rate under the repeated measurements setting. Notably, we develop a novel approximation theory where the constants of the network parameters depend polynomially, rather than exponentially as in existing works, on the dimension, thereby mitigating the curse of dimensionality. Consequently, we derive sharper non-asymptotic bounds for the stochastic error. The finite sample performance of the proposed method is demonstrated through numerical simulations and a real data application.

Problem

• nonparametric regression
• repeated measurements
• covariate shift
• transfer learning
• density ratio

Innovation

• deep regression
• covariate shift
• density ratio estimation
• ReLU neural networks
• minimax optimality