🤖 AI Summary
This work investigates the impact of step-size scheduling on residual dynamics in low-dimensional sparse regression, where excessively fast-decaying learning rates can cause greedy algorithms to suffer from structural stagnation and fail to converge. Focusing on a realizable regression setting with controllable feature coherence, the study integrates greedy approximation theory in Hilbert spaces, coherence analysis, and numerical experiments to systematically examine this phenomenon. The paper unveils, for the first time, the mechanism by which over-decaying step sizes induce structural stagnation and derives an explicit lower bound on the residual norm. Both theoretical analysis and empirical results demonstrate that feature coherence significantly modulates this effect, offering new insights for designing effective step-size schedules in greedy algorithms.
📝 Abstract
Greedy algorithms are central to sparse approximation and stage-wise learning methods such as matching pursuit and boosting. It is known that the Power-Relaxed Greedy Algorithm with step sizes $m^{-\alpha}$ may fail to converge when $\alpha>1$ in general Hilbert spaces. In this work, we revisit this phenomenon from a sparse learning perspective. We study realizable regression problems with controlled feature coherence and derive explicit lower bounds on the residual norm, showing that over-decaying step-size schedules induce structural stagnation even in low-dimensional sparse settings. Numerical experiments confirm the theoretical predictions and illustrate the role of feature coherence. Our results provide insight into step-size design in greedy sparse learning.