Unification and Optimization of Robust Supervised Learning

2026-05-27

AI Summary

Existing robust learning methods are often designed in isolation and struggle to handle unknown dominant failure modes, such as distribution shifts or label noise. This work proposes the first unified framework for robust learning, decomposing approaches into four modular stages: reference distribution augmentation, input perturbation, label perturbation, and sample aggregation—each configurable with pessimistic, neutral, or optimistic strategies. The framework establishes a joint design space encompassing diverse robustness techniques, including distributionally robust optimization, label smoothing, neighborhood risk minimization, and Mixup, enabling end-to-end hyperparameter optimization to adaptively compose optimal strategies. Experiments demonstrate that this approach matches or surpasses the best specialized method across tabular data, image classification, and reward modeling benchmarks, offering a general and reliable default solution for unseen failure modes.

Abstract

The literature has proposed various robust alternatives to empirical risk minimisation to address failure modes such as distribution shift, label noise and finite-sample degeneracies. Examples include distributionally robust optimization, label smoothing, vicinal risk minimization, and Mixup. However, such approaches are typically developed in isolation, forcing practitioners to commit a priori to a single failure mode even when the dominant mode for the task is unclear. To address this, we organize a broad class of existing methods along three common design axes and derive a tractable training procedure that decomposes robust learning into sequential stages (reference distribution enrichment, input-space perturbation, label-space perturbation, and sample-level aggregation), each with a choice of stance (pessimistic, neutral, or optimistic). This results in a unified design space in which joint hyperparameter optimization can compose and configure robustness strategies suited to the task at hand. Across tabular, image, and reward modeling benchmarks, joint hyperparameter optimization is competitive with the best single-method baseline in each setting, offering a reliable default for practitioners who do not know a priori which failure mode dominates their task.

Problem

• robust supervised learning
• distribution shift
• label noise
• finite-sample degeneracies
• empirical risk minimisation

Innovation

• robust supervised learning
• unified design space
• joint hyperparameter optimization
• distribution shift
• label noise