Generalization Error of $f$-Divergence Stabilized Algorithms via Duality

2025-02-20

AI Summary

This paper investigates $f$-divergence-regularized empirical risk minimization (ERM-$f$DR) in constrained optimization settings, addressing the lack of equivalence between its solutions and explicit constraints. We first derive a dual formulation of ERM-$f$DR by leveraging the Legendre–Fenchel transform and the implicit function theorem, enabling explicit derivation of generalization error bounds without relying on strong assumptions such as strong convexity or Lipschitz continuity of loss functions. Under mild regularity conditions, we propose a generic algorithmic framework, establish an explicit generalization bound for the solution, and design an efficient method to compute the normalization constant of the regularized solution. Our core contributions are threefold: (i) unifying constrained optimization and regularization perspectives; (ii) establishing a necessary and sufficient criterion for solution–constraint equivalence; and (iii) achieving a “de-assumptionized” breakthrough in generalization analysis—removing classical structural assumptions while preserving tightness and interpretability.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Constraint Satisfaction and Optimization: Other Foundations of Constraint Satisfaction

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The solution to empirical risk minimization with $f$-divergence regularization (ERM-$f$DR) is extended to constrained optimization problems, establishing conditions for equivalence between the solution and constraints. A dual formulation of ERM-$f$DR is introduced, providing a computationally efficient method to derive the normalization function of the ERM-$f$DR solution. This dual approach leverages the Legendre-Fenchel transform and the implicit function theorem, enabling explicit characterizations of the generalization error for general algorithms under mild conditions, and another for ERM-$f$DR solutions.

Problem

• Extends ERM-$f$DR to constrained optimization problems
• Introduces dual formulation for computational efficiency
• Characterizes generalization error for ERM-$f$DR solutions

Innovation

• Extends ERM-$f$DR to constrained optimization
• Introduces dual formulation for computational efficiency
• Leverages Legendre-Fenchel transform for error analysis