Semiparametric Efficient Bilevel Gradient Estimation

2026-05-20

AI Summary

This work addresses the first-order bias inherent in existing plug-in hypergradient estimators for bilevel optimization with nonparametric lower-level problems. Drawing upon semiparametric efficiency theory, it proposes the first orthogonal hypergradient estimator constructed via the efficient influence function, combined with cross-fitting to eliminate bias and yield an unbiased, asymptotically normal estimate of the population bilevel gradient. Empirical evaluation on synthetic bilevel benchmark tasks demonstrates that the proposed method substantially outperforms both plug-in functional hypergradients and regularized kernel baselines, closely approximating the ideal efficient gradient while ensuring consistent control over outer-level parameters.

Abstract

Functional bilevel methods estimate a lower-level function and plug it into a hypergradient, but this plug-in gradient can retain first-order bias when the lower-level problem is learned nonparametrically. To remove this bias, we develop a semiparametric debiasing theory for population bilevel gradients based on the efficient influence function. This perspective leads to a cross-fitted orthogonal hypergradient estimator for which we establish asymptotic normality together with uniform control over the outer parameter. Under quadratic losses, the estimator reduces to a simple doubly robust score based on conditional mean nuisances. On synthetic bilevel benchmarks with known ground truth, the method tracks the oracle efficient-gradient benchmark and improves over plug-in functional hypergradients and regularized kernel bilevel baselines.

Problem

• bilevel optimization
• gradient estimation
• nonparametric estimation
• bias correction
• semiparametric efficiency

Innovation

• semiparametric debiasing
• bilevel optimization
• efficient influence function
• orthogonal hypergradient
• doubly robust estimation