Feature Priming in Online Linear Regression: Sparse-Regret Lower Bounds and a Tight Univariate Rate

2026-08-18

AI Summary

研究针对高维在线预测中特征稀疏性问题，通过特征预处理方法调整特征权重，并分析了该方法的遗憾界，揭示了干扰插值对预测性能的影响。

Abstract

In high-dimensional online prediction, the best predictor may depend on only a few features, so regret should scale with sparsity rather than the ambient dimension. Feature priming pursues this goal by estimating feature weights from past data and refitting a minimum-norm predictor on the rescaled design. Warmuth and Amid asked at COLT 2023 whether any of three such rules admits a competitive online regret guarantee. Using the natural Moore--Penrose protocol based only on past data, we give a negative answer to the sparse-logarithmic form of this COLT open problem. Our analysis identifies a common obstruction: cheap nuisance interpolation causes the refit to underweight the truly predictive coordinate. An exact target-mass identity and a two-sign argument turn this effect into clipped prediction loss. Hadamard constructions force $Ω(\min\{T,\sqrt{d}\})$ regret for all three rules against a zero-loss one-sparse comparator, with extensions to fixed prime powers and selectors among the rules. Conversely, regret is controlled by data rank, and a Euclidean-normalized triangular construction matches this dependence for powered univariate priming, even under nonnegative second-stage ridge regularization; a paired ridge construction also covers all three powered rules. Exploratory diagnostics on frozen language-model activations exhibit the same relation among nuisance interpolation, target weight, and loss. The exact multivariate and Pearson frontiers remain open.

Problem

• feature priming
• online linear regression
• sparse regret
• high-dimensional prediction
• regret guarantee

Innovation

• Feature Priming
• Sparse-Regret
• Moore--Penrose Protocol
• Nuisance Interpolation
• Hadamard Constructions