Bounded Difference Concentration for Infinitely Exchangeable Sequences with Applications to AI Benchmark Uncertainty

2026-06-15

AI Summary

This work addresses the quantification of uncertainty in accuracy estimation for AI benchmarks such as MMLU that exhibit exchangeable dependency structures. Leveraging de Finetti’s representation theorem, it decomposes the bias of infinitely exchangeable sequences into conditional sampling variability and latent mixture variability. The study reveals, for the first time, a precise cancellation mechanism of the mixture component under zero-sum and linear contrast conditions, yielding a tight Hoeffding-type concentration inequality free of mixture terms. This approach requires no distributional assumptions, applies naturally to both infinite exchangeable sequences and their finite counterparts, and provides domain-stratified upper bounds on uncertainty for composite AI benchmarks. Furthermore, it offers statistical guarantees for low-cost subset evaluations, enabling reliable performance assessment with reduced computational overhead.

Abstract

We consider the concentration properties of functions of infinitely exchangeable random variables. By conditioning on the de Finetti directing measure, we show that the deviation of any function with bounded-difference constants $c_1, \dots, c_n$ decomposes into a conditional sampling fluctuation and a latent mixture fluctuation. When this latent mixture is $σ_{\mathrm{mix}}^2$-subgaussian, we establish a concentration inequality with an effective variance proxy of $\frac{1}{4}\sum_i c_i^2 + σ_{\mathrm{mix}}^2$. Crucially, we demonstrate that for zero-sum linear contrasts, such as the difference between a subsample mean and a full population mean, the latent mixture term cancels exactly. This cancellation yields a tight, mixture-free Hoeffding-type bound that provides a direct de Finetti mechanism for the infinite-extendibility limit of recent finite-exchangeable concentration results. We apply this framework to quantify uncertainty in composite AI benchmarks, such as MMLU, where question items naturally exhibit exchangeable dependence across domains. Our results provide both a domain-stratified hierarchical model for bounding the uncertainty of accuracy scores, and a distribution-free, cost-saving statistical guarantee for accurately estimating full benchmark scores from random subsets.

Problem

• infinitely exchangeable sequences
• concentration inequality
• AI benchmark uncertainty
• bounded difference
• de Finetti theorem

Innovation

• infinitely exchangeable sequences
• bounded difference inequality
• de Finetti representation
• latent mixture cancellation
• AI benchmark uncertainty