Near-Optimal Algorithms for Omniprediction

2025-01-28

AI Summary

This paper addresses the construction of universal predictors: achieving near-optimal prediction performance across all loss functions in a given family ℒ (including Lipschitz and bounded-variation losses) and hypothesis class ℋ, under both online and offline settings. We propose the first online omniprediction framework, attaining an Õ(√T) regret bound—matching the optimal rate for single-loss online learning. We further design an efficient online-to-offline reduction that, for the first time, accommodates infinite hypothesis classes and complex loss families. Our theoretical analysis integrates Rademacher complexity, ERM oracle calls, and loss-specific generalization theory. The offline algorithm outputs an (ℒ, ℋ, ε)-omnipredictor with near-linear sample complexity, where ε is precisely characterized by the Rademacher complexity of the composite function class ℒ ∘ ℋ over ℋ.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Omnipredictors are simple prediction functions that encode loss-minimizing predictions with respect to a hypothesis class $H$, simultaneously for every loss function within a class of losses $L$. In this work, we give near-optimal learning algorithms for omniprediction, in both the online and offline settings. To begin, we give an oracle-efficient online learning algorithm that acheives $(L,H)$-omniprediction with $ ilde{O}(sqrt{T log |H|})$ regret for any class of Lipschitz loss functions $L subseteq L_mathrm{Lip}$. Quite surprisingly, this regret bound matches the optimal regret for emph{minimization of a single loss function} (up to a $sqrt{log(T)}$ factor). Given this online algorithm, we develop an online-to-offline conversion that achieves near-optimal complexity across a number of measures. In particular, for all bounded loss functions within the class of Bounded Variation losses $L_mathrm{BV}$ (which include all convex, all Lipschitz, and all proper losses) and any (possibly-infinite) $H$, we obtain an offline learning algorithm that, leveraging an (offline) ERM oracle and $m$ samples from $D$, returns an efficient $(L_{mathrm{BV}},H,eps(m))$-omnipredictor for $eps(m)$ scaling near-linearly in the Rademacher complexity of $mathrm{Th} circ H$.

Problem

• Optimal Learning
• Prediction Methods
• Bounded Loss Functions

Innovation

• Optimal Learning Method
• Smooth Loss Function
• Online-Offline Learning Adaptability