Weighted-Reward Preference Optimization for Implicit Model Fusion

2024-12-04

arXiv.org

AI Summary

This paper addresses core challenges in fusing heterogeneous open-source large language models (LLMs): inconsistent vocabularies, misaligned parameter matrices, and distributional shift-induced noise accumulation. To this end, we propose WRPO (Weighted Reward Preference Optimization), an implicit fusion method that bypasses explicit vocabulary alignment or parameter merging. Our contributions are threefold: (i) the first preference-optimization-based paradigm for implicit capability transfer; (ii) a progressive adaptation strategy to mitigate distributional shift between source and target models; and (iii) integration of multi-stage policy distillation with weighted reward modeling. On AlpacaEval-2, WRPO built upon LLaMA3-8B-Instruct achieves a 55.9% win rate against GPT-4-Preview-1106 and attains a 46.2% win rate on Arena-Hard versus GPT-4-0314—outperforming existing fusion and fine-tuning approaches by a significant margin.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While fusing heterogeneous open-source LLMs with varying architectures and sizes can potentially integrate the strengths of different models, existing fusion methods face significant challenges, such as vocabulary alignment and merging distribution matrices. These procedures are not only complex but also prone to introducing noise and errors. In this paper, we propose an implicit fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages preference optimization between the source LLMs and the target LLM to transfer their capabilities effectively. WRPO eliminates the need for vocabulary alignment and matrix fusion and can be efficiently scaled to accommodate various LLMs. To address distributional deviations between the source and target LLMs, WRPO introduces a progressive adaptation strategy that gradually shifts reliance on preferred examples from the target LLM to the source LLMs. Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks demonstrate that WRPO consistently outperforms existing knowledge fusion methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct as the target model, WRPO achieves a length-controlled win rate of 55.9% against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2% against GPT-4-0314 on Arena-Hard. Our code is available at https://github.com/SLIT-AI/WRPO.

Problem

• Fuses heterogeneous LLMs without vocabulary alignment.
• Optimizes preference transfer between source and target LLMs.
• Introduces progressive adaptation to reduce distributional deviations.

Innovation

• Weighted-Reward Preference Optimization
• Progressive adaptation strategy
• Eliminates vocabulary alignment