Make Optimization Once and for All with Fine-grained Guidance

2025-03-14

AI Summary

Traditional learned optimization (L2O) methods rely on iterative, local parameter updates, suffering from limited generalization and scalability. To address this, we propose Diff-L2O—the first diffusion-based general-purpose L2O framework—that abandons incremental updates in favor of fine-grained, global solution-space enhancement. Theoretically, we establish the first generalized generalization bound for L2O, quantitatively linking solution diversity to optimization performance. Methodologically, Diff-L2O integrates statistical diversity analysis with a differentiable guidance mechanism, enabling cross-task adaptation within minutes. Empirically, it achieves state-of-the-art performance across diverse optimization tasks—including convex, non-convex, and constrained settings—while reducing training time to minutes (accelerating baselines by over an order of magnitude). Crucially, it requires no task-specific architecture design, significantly improving both generalization capability and practical deployability.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Learning to Optimize (L2O) enhances optimization efficiency with integrated neural networks. L2O paradigms achieve great outcomes, e.g., refitting optimizer, generating unseen solutions iteratively or directly. However, conventional L2O methods require intricate design and rely on specific optimization processes, limiting scalability and generalization. Our analyses explore general framework for learning optimization, called Diff-L2O, focusing on augmenting sampled solutions from a wider view rather than local updates in real optimization process only. Meanwhile, we give the related generalization bound, showing that the sample diversity of Diff-L2O brings better performance. This bound can be simply applied to other fields, discussing diversity, mean-variance, and different tasks. Diff-L2O's strong compatibility is empirically verified with only minute-level training, comparing with other hour-levels.

Problem

• Enhances optimization efficiency with neural networks
• Overcomes limitations of conventional L2O methods
• Introduces Diff-L2O for better generalization and scalability

Innovation

• Diff-L2O framework enhances optimization with diversity
• Generalization bound improves performance across tasks
• Minute-level training outperforms hour-level methods