Realizable Bayes-Consistency for General Metric Losses

2026-05-05

AI Summary

This work investigates the conditions under which strong Bayesian consistency—defined as almost sure convergence of risk to zero—holds in the realizable setting when both instance and label spaces are equipped with a general metric loss. To this end, the authors introduce a novel combinatorial obstruction structure, termed an infinite non-decreasing $(\gamma_k)$-Littlestone tree, and develop a learnability framework for general metric losses by leveraging extended Littlestone trees, metric space theory, and tools from real analysis. This framework establishes necessary and sufficient conditions for a hypothesis class to admit strong Bayesian consistency, thereby fully characterizing the existence of distribution-independent learning rules and resolving an open problem posed by Cohen et al.

Abstract

We study strong universal Bayes-consistency in the realizable setting for learning with general metric losses, extending classical characterizations beyond $0$-$1$ classification \citep{bousquet_theory_2021, hanneke2021universalbayesconsistencymetric} and real-valued regression \citep{attias_universal_2024}. Given an instance space $(\mathcal X,ρ)$, a label space $(\mathcal Y,\ell)$ with possibly unbounded loss, and a hypothesis class $\mathcal H \subseteq \mathcal Y^{\mathcal X}$, we resolve the realizable case of an open problem presented in \citet{pmlr-v178-cohen22a}. Specifically, we find the necessary and sufficient conditions on the hypothesis class $\mathcal H$ under which there exists a distribution-free learning rule whose risk converges almost surely to the best-in-class risk (which is zero) for every realizable data-generating distribution. Our main contribution is this sharp characterization in terms of a combinatorial obstruction: Similarly to \citet{attias2024optimallearnersrealizableregression}, we introduce the notion of an infinite non-decreasing $(γ_k)$-Littlestone tree, where $γ_k \to \infty$. This extends the Littlestone tree structure used in \citet{bousquet_theory_2021} to the metric loss setting.

Problem

• Bayes-consistency
• metric losses
• realizable setting
• hypothesis class
• distribution-free learning

Innovation

• Bayes-consistency
• metric loss
• Littlestone tree
• realizable setting
• distribution-free learning