Fine-tuning Factor Augmented Neural Lasso for Heterogeneous Environments

2026-04-14

AI Summary

This work addresses the challenges of variable selection in high-dimensional nonparametric regression and transfer learning under covariate or posterior shift. It proposes the FAN-Lasso framework, which models high-dimensional correlated covariates via a low-rank factor structure and introduces a residual fine-tuning decomposition that expresses the target function as a combination of a frozen source function and an adaptable variable transformation, thereby enabling knowledge transfer and nonparametric variable selection. The study establishes, for the first time, a theoretical framework for fine-tuning in high-dimensional nonparametric settings, revealing how sample size and function complexity jointly influence statistical acceleration and providing theoretical justification for parameter-efficient adaptation. Empirical results demonstrate that the method significantly outperforms baselines across diverse shift scenarios and achieves near-oracle performance even with extremely limited target samples, corroborating the tightness of the derived risk bounds.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Fine-tuning is a widely used strategy for adapting pre-trained models to new tasks, yet its methodology and theoretical properties in high-dimensional nonparametric settings with variable selection have not yet been developed. This paper introduces the fine-tuning factor augmented neural Lasso (FAN-Lasso), a transfer learning framework for high-dimensional nonparametric regression with variable selection that simultaneously handles covariate and posterior shifts. We use a low-rank factor structure to manage high-dimensional dependent covariates and propose a novel residual fine-tuning decomposition in which the target function is expressed as a transformation of a frozen source function and other variables to achieve transfer learning and nonparametric variable selection. This augmented feature from the source predictor allows for the transfer of knowledge to the target domain and reduces model complexity there. We derive minimax-optimal excess risk bounds for the fine-tuning FAN-Lasso, characterizing the precise conditions, in terms of relative sample sizes and function complexities, under which fine-tuning yields statistical acceleration over single-task learning. The proposed framework also provides a theoretical perspective on parameter-efficient fine-tuning methods. Extensive numerical experiments across diverse covariate- and posterior-shift scenarios demonstrate that the fine-tuning FAN-Lasso consistently outperforms standard baselines and achieves near-oracle performance even under severe target sample size constraints, empirically validating the derived rates.

Problem

• fine-tuning
• high-dimensional nonparametric regression
• variable selection
• covariate shift
• posterior shift

Innovation

• fine-tuning
• factor augmentation
• neural Lasso
• transfer learning
• minimax optimality