Beyond Short Steps in Frank-Wolfe Algorithms

2025-01-30

AI Summary

Frank–Wolfe (FW) algorithms suffer from conservative step sizes and lack rigorous primal–dual coordination guarantees. Method: We propose the first optimistic FW algorithm with a provably convergent primal–dual framework. It introduces an optimistic prediction–correction mechanism and a generalized short-step strategy, extending gap-driven adaptive updates—previously limited to primal-only methods—to gradient-based algorithms. By integrating smoothness exploitation and convergence-rate tightening techniques, our method ensures computable primal–dual gaps and verifiable step sizes. Contribution/Results: We establish the tightest known $O(1/k)$ primal–dual convergence rate for FW-type methods. Experiments on structured convex optimization tasks—including Lasso, matrix completion, and Wasserstein distance computation—demonstrate that our algorithm significantly outperforms classical FW and existing variants in accuracy, speed, and practicality.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We introduce novel techniques to enhance Frank-Wolfe algorithms by leveraging function smoothness beyond traditional short steps. Our study focuses on Frank-Wolfe algorithms with step sizes that incorporate primal-dual guarantees, offering practical stopping criteria. We present a new Frank-Wolfe algorithm utilizing an optimistic framework and provide a primal-dual convergence proof. Additionally, we propose a generalized short-step strategy aimed at optimizing a computable primal-dual gap. Interestingly, this new generalized short-step strategy is also applicable to gradient descent algorithms beyond Frank-Wolfe methods. As a byproduct, our work revisits and refines primal-dual techniques for analyzing Frank-Wolfe algorithms, achieving tighter primal-dual convergence rates. Empirical results demonstrate that our optimistic algorithm outperforms existing methods, highlighting its practical advantages.

Problem

• Frank-Wolfe Algorithm
• Step Size Selection
• Stopping Conditions Optimization

Innovation

• Frank-Wolfe Algorithm
• Step Size Optimization
• Primal-Dual Gap Reduction