Clipping the Price of Adaptivity at the Tail

2026-06-21

AI Summary

This work addresses the challenge that existing adaptive stochastic convex optimization methods struggle to efficiently handle large uncertainties in both the initial error and the Lipschitz constant simultaneously. The paper proposes a novel approach that decomposes the objective function into model and loss components and applies clipping to the model outputs under tail events, steering them toward those of a reference model. This method is the first to achieve efficient adaptation to dual uncertainties—initial distance and Lipschitz constant—while preserving the structural properties typical of learning problems. The resulting convergence bound incurs only a logarithmic factor overhead compared to the optimal bound attainable when these parameters are known a priori, thereby overcoming the prevailing bottleneck in adaptivity costs.

Abstract

Adaptive stochastic convex optimization (SCO) methods face a fundamental ``price of adaptivity'' barrier: under the standard set of assumptions, they cannot efficiently adapt to large uncertainty in both the initial distance to optimality and the Lipschitz constant. We circumvent this barrier by requiring a small amount of additional structure common to many learning problems. Specifically, we assume that the objective decomposes into a model and a loss function, enabling us to intervene by modifying the model's output before it passes to the loss function. Under this assumption, we design a method that clips the learned model output in tail events where it deviates too much from the output of a fixed reference model. Our method matches the optimal bounds for known-parameter SCO up to logarithmic factors in the uncertainty in the distance and Lipschitz parameters, thus efficiently adapting to large uncertainty in both.

Problem

• adaptive stochastic convex optimization
• price of adaptivity
• Lipschitz constant
• initial distance to optimality
• uncertainty

Innovation

• adaptive stochastic optimization
• price of adaptivity
• output clipping
• tail event handling
• structured SCO