Dangerous Liaisons of Convex Learning and Non-Affine Aggregation

2026-06-26

AI Summary

This work addresses the detrimental effects of non-affine gradient aggregation in convex optimization, which disrupts the monotonicity of update operators and consequently undermines algorithmic convergence and generalization. We establish, for the first time, that only positive affine aggregations preserve monotonicity, and we derive sufficient conditions to restore it. Leveraging monotone operator theory, convex optimization, and algorithmic stability analysis, we rigorously quantify how non-affine aggregation degrades both convergence and stability. Our unified theoretical framework elucidates the common failure modes of modern learning systems under constraints related to adaptivity, privacy, robustness, and fairness, thereby providing principled guidance for designing algorithms that simultaneously satisfy such constraints and guarantee convergence.

Abstract

Last-iterate convergence and generalization guarantees in first-order convex learning hinge on the monotonicity of the update operator. While linear averaging preserves the monotonicity of gradient updates, this property is often violated when gradients are aggregated non-affinely, as in modern pipelines enforcing constraints like adaptivity, privacy, robustness or fairness. Whether it is possible to design non-affine aggregation rules that maintain monotonicity has remained an open question. We answer this question negatively: we prove that the monotonicity of aggregated gradients is preserved if and only if the aggregation rule is positively affine. Consequently, non-affine aggregation prevents steady convergence and substantially degrade algorithmic stability. We quantify these drawbacks and propose a path forward by identifying sufficient conditions under which monotonicity can be restored. Our results provide a unified theoretical framework explaining the disparate failure modes observed in modern learning systems.

Problem

• convex learning
• non-affine aggregation
• monotonicity
• last-iterate convergence
• algorithmic stability

Innovation

• monotonicity
• non-affine aggregation
• convex learning
• last-iterate convergence
• algorithmic stability