Tuning-Free Efficient Estimation for Multi-Source Data via Covariance-Aware Shrinkage

2026-06-29

AI Summary

This work addresses the trade-off between bias control and learning efficiency in multi-source heterogeneous data settings by proposing a parameter-free, covariance-aware sequential shrinkage framework. The method adaptively constructs shrinkage directions based on the covariance structure of source data and integrates multiple sources sequentially to maximize risk reduction under limited sample sizes. It achieves, for the first time, a fully data-driven, tuning-free multi-source shrinkage estimator, establishes non-asymptotic risk bounds, and proves that the proposed strategy asymptotically attains the oracle risk. Both theoretical analysis and empirical experiments demonstrate that, particularly in highly heterogeneous scenarios, the approach significantly outperforms existing methods in terms of estimation accuracy and computational efficiency.

Abstract

Modern statistical learning problems often involve multiple related data sets, where learning efficiency on a target set can be improved by utilizing related source sets, while heterogeneity among the source sets may introduce bias. Existing approaches are limited by suboptimal performance in multi-source settings, insufficient use of covariance information, or the computational burden of tuning procedures. We propose a tuning-free and covariance-aware shrinkage framework that constructs shrinkage directions using covariance information to improve efficiency. We establish finite-sample risk bounds that yield an explicit risk-improving interval for the shrinkage size, making the procedure fully data-driven and tuning-free. When multiple source sets are available, we further propose a novel sequential algorithm that shrinks the estimator toward the sources one at a time according to their estimated risk reduction. The proposed algorithm asymptotically attains the oracle risk under mild conditions and is guaranteed to improve over the single-step shrinkage method in the literature. The framework is further extended to general smooth \(M\)-estimation problems via a local quadratic approximation. Numerical studies show substantial gains over competing methods, especially when the source data sets are highly heterogeneous.

Problem

• multi-source data
• heterogeneity
• covariance-aware
• tuning-free
• estimation efficiency

Innovation

• covariance-aware shrinkage
• tuning-free estimation
• multi-source data
• sequential shrinkage algorithm
• M-estimation