Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging

2024-08-22

AI Summary

Existing model merging methods produce only a single merged model, failing to accommodate diverse performance preferences across base models—leading to suboptimal trade-offs and insufficient personalization. This work proposes a preference-aware multi-objective model merging paradigm, formulating merging as a multi-objective optimization problem where each base model’s task-specific performance serves as an objective. For the first time, it enables *single-shot* generation of a Pareto-optimal solution set, allowing users to select customized models on-demand. Our approach leverages parameter-efficient fine-tuning architectures, integrating gradient projection with Pareto front search. Extensive experiments across multiple benchmarks demonstrate that the generated model sets significantly outperform state-of-the-art merging methods: they maintain competitive overall performance while substantially enhancing model diversity and personalization capability.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Metareasoning and Metaheuristics
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Model merging, which combines multiple models into a single model, has gained popularity in recent years. By efficiently integrating the capabilities of various models, this significantly reduces the parameter count and memory usage. However, current methods can only produce one single merged model. This necessitates a performance trade-off due to conflicts among the various models, and the resultant one-size-fits-all model may not align with the preferences of different users who may prioritize certain models over others. To address this issue, we propose preference-aware model merging, and formulate this as a multi-objective optimization problem in which the performance of the merged model on each base model's task is treated as an objective. In a single merging process, the proposed parameter-efficient structure generates a Pareto set of merged models, with each representing a Pareto-optimal solution for a preference. Users can then select merged models tailored to their preferences from this learned Pareto set. Experimental results demonstrate that the proposed Pareto Merging produces diverse trade-off models and achieves higher test accuracy compared to state-of-the-art merging baselines.

Problem

• Preference-aware model merging for diverse user needs
• Multi-objective optimization to balance model performance
• Generating Pareto-optimal models for tailored preferences

Innovation

• Preference-aware model merging
• Multi-objective optimization framework
• Pareto set generation