A Dual Optimization View to Empirical Risk Minimization with f-Divergence Regularization

2025-08-05

AI Summary

In joint optimization of empirical risk minimization (ERM) and $f$-divergence regularization ($f$-DR), the normalization function lacks a closed-form solution, hindering scalability and efficiency. Method: We propose a dual-theoretic framework grounded in Legendre–Fenchel duality and the implicit function theorem, rigorously linking the primal and dual problems. This analysis yields a nonlinear ordinary differential equation (ODE) characterizing the normalization function, which admits efficient numerical integration under mild regularity conditions. Contribution/Results: The method enables rapid, accurate computation of the normalization function, significantly enhancing the scalability and computational efficiency of ERM-$f$-DR models. It establishes a novel theoretical pathway for $f$-divergence-regularized learning and provides a practical numerical tool. Empirical and theoretical validation confirms the effectiveness and broad applicability of dual analysis in statistical learning.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

The dual formulation of empirical risk minimization with f-divergence regularization (ERM-fDR) is introduced. The solution of the dual optimization problem to the ERM-fDR is connected to the notion of normalization function introduced as an implicit function. This dual approach leverages the Legendre-Fenchel transform and the implicit function theorem to provide a nonlinear ODE expression to the normalization function. Furthermore, the nonlinear ODE expression and its properties provide a computationally efficient method to calculate the normalization function of the ERM-fDR solution under a mild condition.

Problem

• Dual optimization for empirical risk minimization with f-divergence regularization
• Connecting dual solution to implicit normalization function via ODE
• Efficient computation of normalization function under mild conditions

Innovation

• Dual optimization with f-divergence regularization
• Legendre-Fenchel transform for nonlinear ODE
• Implicit function theorem for efficient computation