Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data

2026-07-07

AI Summary

This work investigates the impact of memoryless stability annealing on the implicit bias and convergence trajectory of smoothed sign descent in linear classification with separable data. By precisely reformulating the dynamics as entropy mirror ascent on a concave dual objective, the study establishes—for the first time—that normalized iterates converge to the minimizer of a convex Burg-type barrier function over the margin slice, and fully characterizes its static geometric structure and endpoint asymptotics. The analysis combines dual gap control, recursive KL divergence bounds, and envelope analysis of normalized iterates. Theoretical predictions align closely with experiments, verifying dual identities up to floating-point precision and revealing convergence trajectories, rates, and cross-scaling phenomena at fixed ε.

Abstract

Adaptive gradient methods can favor max-margin separators that differ from gradient descent, yet a fixed positive numerical stability constant eventually changes the update geometry again. This paper studies the rate-controlled middle case for full-batch linear classification on separable data. For memoryless stability-annealed smoothed-sign descent with weighted exponential loss, we prove that the normalized iterates converge to the minimizer of a convex Burg-type barrier over a margin slice. The proof rewrites the dynamics exactly as entropic mirror ascent on a concave dual objective, controls the dual gap by a KL recursion, and yields an explicit S_t^{-1/2} normalized-iterate envelope. The static barrier geometry is fully characterized, including KKT conditions and both endpoint limits. Experiments validate the exact dual identities to floating-point error, illustrate the predicted path and rate diagram, and show an empirical fixed-epsilon crossover scaling in cumulative time. We further report robustness and boundary diagnostics for logistic tails, fixed-epsilon crossover, and adaptive-method variants, delineating the scope of the proved smoothed-sign theory.

Problem

• implicit bias
• smoothed sign descent
• stability annealing
• separable data
• adaptive gradient methods

Innovation

• stability annealing
• smoothed-sign descent
• implicit bias
• entropic mirror ascent
• Burg-type barrier