Differentially Private Bilevel Optimization: Efficient Algorithms with Near-Optimal Rates

2025-06-15

AI Summary

This work studies differentially private bilevel optimization, motivated by hierarchical learning applications—such as meta-learning and hyperparameter optimization—where sensitive data are involved. We propose the first unified privacy-preserving framework applicable to both convex and nonconvex bilevel structures. Theoretically, we establish the first near-optimal lower bound on the privacy–utility trade-off. Methodologically, we design a log-concave sampling mechanism based on inexact function evaluations, circumventing explicit gradient computation. Algorithmically, for nonconvex settings, our method achieves iteration complexity independent of the inner-level dimension. Experimentally, under convex assumptions, it attains the optimal excess risk bound matching those of private ERM and SCO; under nonconvex assumptions, it achieves the state-of-the-art convergence rate to ε-approximate stationary points. Overall, our approach significantly improves the synergy between privacy protection and computational efficiency.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Security and privacy of machine learning and AI applications
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

Bilevel optimization, in which one optimization problem is nested inside another, underlies many machine learning applications with a hierarchical structure -- such as meta-learning and hyperparameter optimization. Such applications often involve sensitive training data, raising pressing concerns about individual privacy. Motivated by this, we study differentially private bilevel optimization. We first focus on settings where the outer-level objective is extit{convex}, and provide novel upper and lower bounds on the excess risk for both pure and approximate differential privacy, covering both empirical and population-level loss. These bounds are nearly tight and essentially match the optimal rates for standard single-level differentially private ERM and stochastic convex optimization (SCO), up to additional terms that capture the intrinsic complexity of the nested bilevel structure. The bounds are achieved in polynomial time via efficient implementations of the exponential and regularized exponential mechanisms. A key technical contribution is a new method and analysis of log-concave sampling under inexact function evaluations, which may be of independent interest. In the extit{non-convex} setting, we develop novel algorithms with state-of-the-art rates for privately finding approximate stationary points. Notably, our bounds do not depend on the dimension of the inner problem.

Problem

• Developing differentially private bilevel optimization algorithms
• Achieving near-optimal rates for convex and non-convex objectives
• Addressing privacy concerns in hierarchical machine learning applications

Innovation

• Differentially private bilevel optimization algorithms
• Efficient log-concave sampling with inexact evaluations
• Dimension-independent non-convex stationary point methods