Generalization Error of the Tilted Empirical Risk

2024-09-28

arXiv.org

AI Summary

This work investigates the generalization error and robustness of tilted empirical risk (TER) under negative tilting regimes. For the distributionally invariant setting, we establish the first unified information-theoretic analytical framework, deriving a uniform $O(1/sqrt{n})$ upper bound and an $O(1/n)$ expected convergence rate for tilted generalization error—achieved via KL regularization—thereby overcoming fundamental limitations of conventional linear risk measures. Building upon exponential tilting transformations, entropy-based bounds, and empirical process theory, we obtain tight convergence-rate upper bounds and rigorously prove their superiority over standard empirical risk minimization in convergence speed. Our results provide novel theoretical foundations and an analytical paradigm for tilted learning and robust statistical learning.

Technology Category

• Machine Learning: Learning Theory
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

The generalization error (risk) of a supervised statistical learning algorithm quantifies its prediction ability on previously unseen data. Inspired by exponential tilting, Li et al. (2021) proposed the tilted empirical risk as a non-linear risk metric for machine learning applications such as classification and regression problems. In this work, we examine the generalization error of the tilted empirical risk. In particular, we provide uniform and information-theoretic bounds on the tilted generalization error, defined as the difference between the population risk and the tilted empirical risk, with a convergence rate of $O(1/sqrt{n})$ where $n$ is the number of training samples. Furthermore, we study the solution to the KL-regularized expected tilted empirical risk minimization problem and derive an upper bound on the expected tilted generalization error with a convergence rate of $O(1/n)$.

Problem

• Analyzing generalization error of tilted empirical risk under negative tilt
• Providing bounds for tilted generalization error with unbounded loss functions
• Studying robustness of tilted empirical risk against noisy outliers

Innovation

• Proposes tilted empirical risk for robustness
• Provides bounds for tilted generalization error
• Studies robustness against noisy outliers