Simple Optimizers for Convex Aligned Multi-Objective Optimization

2025-09-06

AI Summary

Existing alignment-based multi-objective optimization (AMOO) theory relies on strong convexity—ensuring a unique global optimum—but this assumption rarely holds in deep learning. Method: We present the first systematic analysis of gradient descent for convex AMOO under standard smoothness or Lipschitz continuity assumptions, *without* requiring strong convexity. We introduce novel analytical tools and a scalable optimizer tailored to AMOO. Contribution/Results: We rigorously prove the inherent suboptimality of naive equal-weighting, and establish the first convergence rate upper bound and information-theoretic lower bound for non-strongly convex convex AMOO. Our theoretical results demonstrate that the proposed method significantly outperforms equal-weight baselines while maintaining scalability. Empirical evaluation confirms consistent improvements in optimization efficiency and generalization across multi-task learning benchmarks, providing a more rigorous and practical theoretical foundation and algorithmic framework for deep multi-task optimization.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

It is widely recognized in modern machine learning practice that access to a diverse set of tasks can enhance performance across those tasks. This observation suggests that, unlike in general multi-objective optimization, the objectives in many real-world settings may not be inherently conflicting. To address this, prior work introduced the Aligned Multi-Objective Optimization (AMOO) framework and proposed gradient-based algorithms with provable convergence guarantees. However, existing analysis relies on strong assumptions, particularly strong convexity, which implies the existence of a unique optimal solution. In this work, we relax this assumption and study gradient-descent algorithms for convex AMOO under standard smoothness or Lipschitz continuity conditions-assumptions more consistent with those used in deep learning practice. This generalization requires new analytical tools and metrics to characterize convergence in the convex AMOO setting. We develop such tools, propose scalable algorithms for convex AMOO, and establish their convergence guarantees. Additionally, we prove a novel lower bound that demonstrates the suboptimality of naive equal-weight approaches compared to our methods.

Problem

• Relaxing strong convexity assumptions in multi-objective optimization
• Developing gradient-descent algorithms under standard smoothness conditions
• Addressing suboptimality of naive equal-weight optimization approaches

Innovation

• Relaxed convexity assumptions for AMOO
• Proposed scalable gradient-descent algorithms
• Developed new convergence metrics and tools