Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generation

2025-02-15

AI Summary

Existing methods struggle to simultaneously achieve multi-objective controllable generation and rapid adaptation to user preferences during inference, primarily due to neglecting how inter-objective competition affects fine-tuning. To address this, we propose the “Bone-Seeking–Broth-Simmering” paradigm: first, multiple specialized backbone models are jointly trained via multi-objective reinforcement learning; second, a Pareto-optimal and controllable model merging framework is constructed—where backbone rewards are derived from basis vectors and interpretable, dynamically adaptive weighting coefficients are generated via symmetric circulant matrices—enabling real-time, preference-driven fusion. This work is the first to unify Pareto frontier optimization with structured parameter merging within a controllable generation framework. Experiments on multi-objective text generation demonstrate significant improvements in Pareto optimality, controllability, and inference-time adaptation efficiency.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Generation
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

User information needs are often highly diverse and varied. A key challenge in current research is how to achieve controllable multi-objective generation while enabling rapid adaptation to accommodate diverse user demands during test time. Existing solutions, such as Rewarded Soup, focus on merging language models individually tuned on single objectives. While easy to implement and widely used, these approaches face limitations in achieving optimal performance due to their disregard for the impacts of competing objectives on model tuning. To address this issue, we propose Bone Soup, a novel model merging approach that first seeks a series of backbone models by considering the impacts of multiple objectives and then makes the soup (i.e., merge the backbone models). Specifically, Bone Soup begins by training multiple backbone models for different objectives using multi-objective reinforcement learning. Each backbone model is guided by a combination of backbone reward signals. To ensure that these models are optimal for the Pareto front, the backbone rewards are crafted by combining standard reward functions into basis vectors, which can then be modified through a rule-based construction method. Bone Soup leverages a symmetric circulant matrix mapping to generate the merging coefficients, which are used to merge the backbone models according to user preferences. Extensive experimental results demonstrate that Bone Soup exhibits strong controllability and Pareto optimality in controllable multi-objective generation, providing a more effective and efficient approach to addressing diverse user needs at test time.

Problem

• Achieving controllable multi-objective generation
• Adapting to diverse user demands
• Optimizing model merging for competing objectives

Innovation

• Multi-objective reinforcement learning
• Symmetric circulant matrix mapping
• Rule-based reward construction