Retrieval-Guided Fine-Tuning as Noisy Estimation: Risk bounds and Architectural Analysis

2026-09-13

AI Summary

研究通过多任务线性回归框架分析检索引导微调(RAG-FT)中的噪声估计问题，提出距离比例噪声模型，并比较不同估计方法的风险。

Abstract

Retrieval-Guided Fine-Tuning (RAG-FT) incorporates retrieved data directly into the training objective, but the statistical consequences of noisy retrieval during training remain theoretically undercharacterized. We study this question by modeling RAG-FT as an estimation problem in a multi-task linear regression framework, using an OLS proxy for single-layer linear self-attention to obtain finite-sample risk bounds. Under homoscedastic retrieval noise, we show that retrieval failure decays exponentially with task separation relative to noise, and derive explicit finite-sample conditions under which RAG-FT achieves lower risk than both target-only and full-corpus training. We then introduce a Distance-Proportional Noise (DPN) model, in which retrieval quality degrades with rank, and compare two estimators under the same retrieval process: the OLS proxy and the literal, uniform-weight forward pass of linear self-attention. We prove that the attention estimator's bias diverges as $Θ(n^{2q})$ even under exact retrieval, while OLS risk remains $Θ(d/n)$ for every noise exponent $q>0$. These results locate the instability not in noisy retrieval itself, but in the fixed, unweighted aggregation of the literal LSA forward pass, which reweighting by reliability empirically removes. We validate the predicted rate separation through direct simulation of the DPN model.

Problem

• Retrieval-Guided Fine-Tuning
• noisy retrieval
• statistical consequences
• estimation problem
• multi-task linear regression

Innovation

• Retrieval-Guided Fine-Tuning
• multi-task linear regression
• Distance-Proportional Noise model
• Ordinary Least Squares proxy