Logarithmic-Free Moment and Generalization Bounds for Uniformly Stable Algorithms

2026-08-10

AI Summary

This work addresses the long-standing issue of an extraneous $\log n$ factor in generalization error bounds for uniformly stable algorithms. By establishing a moment inequality for sums of weakly interacting functions, the authors derive the first tight upper bound within the uniform stability framework that eliminates the logarithmic term entirely. Their approach combines Rademacher complexity estimates with a double-sample randomization technique and extends to arbitrary product distributions. The main result shows that, for any $p \geq 2$, the $L_p$ norm of the function sum is bounded by $16pn\beta + M\sqrt{2pn}$, thereby removing the $\log n$ factor and matching known lower bounds—resolving an open problem in the field.

Abstract

Uniform stability is a classical tool for controlling the generalization error of a learning algorithm. Bousquet, Klochkov, and Zhivotovskiy (2020) showed that the problem can be reduced to a moment inequality for a sum of weakly interacting functions of independent random variables. Their bound contains an additional factor $\log n$, and they asked whether this factor can be removed. We answer this upper-bound question affirmatively. More specifically, let $Z=(Z_1,\ldots,Z_n)$ have independent coordinates and let $g_i(Z)$ satisfy $$ \mathbb E[g_i(Z)\mid Z_{-i}]=0, \qquad \left| \mathbb E[g_i(Z)\mid Z_i]\right|\le M, \qquad \forall i = \overline{1, n} $$ while changing any coordinate $Z_j$, $j\neq i$, changes $g_i$ by at most $β$ and $Z_{-i}$ denotes all coordinates except $Z_i$. We prove that, for every $p\ge2$, $$ \left\| \sum_{i=1}^n g_i(Z)\right\|_p \le 16pnβ+M\sqrt{2pn}. $$ This removes the $\log n$ factor from the previous bound and matches the lower bound of Bousquet, Klochkov, and Zhivotovskiy up to universal constants in the range covered by their construction. Our proof first establishes the required estimate on the Rademacher cube, then transfers it to arbitrary product distributions by a two-copy randomization argument.

Problem

• uniform stability
• generalization bounds
• moment inequalities
• logarithmic factor
• learning algorithms

Innovation

• uniform stability
• moment bounds
• generalization error
• logarithmic-free
• Rademacher complexity