Aggregation with Exponential Weights is Optimal in Expectation

2026-07-02

AI Summary

This study resolves a long-standing open question regarding whether Aggregation with Exponential Weights (AEW) can achieve minimax-optimal rates in expectation under random design and squared loss. Focusing on model selection aggregation, the authors establish—for the first time—that AEW attains an expected excess risk of order \( T\log M/(n+1) \), thereby achieving minimax optimality, under the assumptions that the loss is strongly convex, Lipschitz continuous, and bounded, without requiring the Bernstein condition. This result holds when the temperature parameter is fixed and satisfies \( T \geq 4b^2 \), where predictions and labels lie in \([0,b]\). The analysis further reveals a sharp phase transition in AEW’s performance as a function of the temperature parameter.

Abstract

The aggregation with exponential weights (AEW) estimator is not fully understood in the basic setting of model selection aggregation with squared loss. In particular, whether it is minimax-rate optimal in expectation for large enough fixed temperatures and under random design has been an open problem since its introduction, which was explicitly posed by Lecué and Mendelson (2013). In this paper, we settle this problem by showing that \emph{without} requiring a Bernstein-type assumption, the AEW indeed achieves the excess risk $T \log (M) / (n+1)$ in expectation, whenever the temperature $T$ satisfies $(L^2/T)\exp(B/T)\leq μ/2$. Here, the number of dictionary elements is $M$, the estimator has observed $n$ i.i.d. samples from any distribution, and the loss is assumed to be bounded by $B$, $L$-Lipschitz continuous and $μ$-strongly convex. For squared loss, we show that $T\geq 4 b^2$ suffices when the predictions and labels are $[0,b]$-valued. Because AEW is known to be suboptimal in expectation for temperatures below some constant, this shows that AEW has a sharp phase transition when the temperature is large enough but constant, as conjectured by Lecué and Mendelson.

Problem

• aggregation with exponential weights
• model selection
• minimax optimality
• excess risk
• random design

Innovation

• Aggregation with Exponential Weights
• minimax optimality
• model selection aggregation
• phase transition
• excess risk