Functional Gradient Descent with Adaptive Representations

2026-06-15

AI Summary

This work addresses the challenges in function space optimization arising from non-convex losses induced by fixed representations and the intractability of infinite-dimensional gradients. To overcome these issues, the authors propose a practical adaptive functional gradient descent method that dynamically adjusts the function representation during optimization. By integrating gradient approximation in function spaces with rigorous convergence analysis, the method establishes, for the first time under general settings, a computationally feasible algorithm with theoretical guarantees of convergence—either to a stationary point or the global optimum—relying on smoothness and Polyak–Łojasiewicz-type conditions. Empirical evaluations demonstrate consistent superiority over both fixed-representation functional gradient descent and neural network baselines across diverse tasks, including regression, partial differential equation solving, and computer vision, achieving higher accuracy and computational efficiency.

Abstract

Functional optimization problems are typically solved by optimizing the parameters of a fixed representation, such as a neural network, resulting in highly nonconvex losses that complicate both training and theoretical analysis. An interesting alternative is functional gradient descent (FGD), that is, gradient descent directly in function space, which benefits from strong convergence results and admits a clean theory. However, FGD is difficult to implement in practice because functional gradients are infinite-dimensional, and thus cannot be fully computed nor stored in memory. Existing implementations therefore rely on fixed approximations, which introduce approximation error. We propose a new, theoretically-grounded FGD algorithm that adapts the representation of the functional gradients over the course of optimization. By explicitly incorporating this approximation into the analysis, we establish convergence to a stationary point (for smooth losses) and to a global minimizer (under smoothness + a Polyak-Lojasiewicz-type condition) regardless of our approximations. To the best of our knowledge, this is the first implementable FGD method with such guarantees in a general setting. We demonstrate the effectiveness of our method on regression, numerical solution of PDEs, and modern computer vision. Across settings, our method consistently outperforms both FGD with fixed approximations and neural network baselines in efficiency and accuracy.

Problem

• functional optimization
• functional gradient descent
• infinite-dimensional gradients
• approximation error
• nonconvex optimization

Innovation

• Functional Gradient Descent
• Adaptive Representations
• Convergence Guarantees
• Function Space Optimization
• Polyak-Łojasiewicz Condition