Federated stochastic bilevel optimization with fully first-order gradients

2026-09-14

AI Summary

本文针对联邦随机双层优化中计算二阶矩阵导致运行时间长的问题，提出了一种仅依赖一阶梯度的新算法，并引入了新的学习率机制来提高效率。

Abstract

Federated stochastic bilevel optimization has been actively studied in recent years due to its widespread applications in machine learning. However, most existing federated stochastic bilevel optimization algorithms require the computation of second-order Hessian and Jacobian matrices, which leads to longer running times in practice. To address these challenges, we propose a novel federated stochastic variance-reduced bilevel gradient descent algorithm that relies solely on first-order oracles. Specifically, our approach does not require the computation of second-order Hessian and Jacobian matrices, significantly reducing running time. Furthermore, we introduce a novel learning rate mechanism, i.e., a constant single-timescale learning rate, to coordinate the update of different variables. We also present a new strategy to establish the convergence rate of our algorithm. Finally, the extensive experimental results confirm the efficacy of our proposed algorithm.

Problem

• Federated stochastic bilevel optimization
• second-order Hessian
• Jacobian matrices

Innovation

• federated stochastic bilevel optimization
• first-order oracles
• constant single-timescale learning rate