H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs

2024-11-26

arXiv.org

AI Summary

This work addresses the limited performance gains and insufficient robustness in multi-model alignment fusion. We propose H³Fusion, a novel method built upon a Mixture-of-Experts (MoE) architecture, featuring an instruction-aware dynamic expert routing mechanism that fine-tunes only the feed-forward network (FFN) layers while keeping attention weights frozen. To the best of our knowledge, H³Fusion introduces the first triple-alignment fusion paradigm, jointly optimizing helpfulness, harmlessness, and honesty via a gated loss function and adaptive regularization—thereby co-regulating expert activation distributions and parameter drift. Evaluated on three core alignment benchmarks, H³Fusion achieves an average 11.37% improvement in alignment performance and demonstrates 13.77% higher robustness than the current state-of-the-art ensemble methods, significantly enhancing model outputs’ helpfulness, harmlessness, and honesty.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Alignment of pretrained LLMs using instruction-based datasets is critical for creating fine-tuned models that reflect human preference. A growing number of alignment-based fine-tuning algorithms and benchmarks emerged recently, fueling the efforts on effective alignments of pre-trained LLMs to ensure helpful, harmless, and honest answers from both open-source and closed-source LLMs. This paper tackles this problem by developing an alignment fusion approach, coined as $H^3$Fusion, with three unique characteristics. First, $H^3$Fusion ensembles multiple individually aligned LLMs to create a final fine-tuned alignment model with enhanced capabilities beyond those of individual models, delivering robust alignment through promoting helpful, harmless, honest fusion. Second, $H^3$Fusion leverages the mixture-of-experts (MoE) methodology in two steps. We first freeze the multi-head attention weights of each individual model while tuning the FFN layer during alignment fusion. Then we merge the aligned model weights with an expert router according to the type of input instruction and dynamically select a subset of experts that are best suited for producing the output response. Finally, we boost the performance of the resulting $H^3$3Fusion model by introducing gating loss and regularization terms. The former penalizes the selection errors of the expert-router, and the latter mediates the expert weights drifting during fine-tuning and dynamically adjusts the fusion behavior of the resulting model by canalizing the activations on the experts. Extensive evaluations on three benchmark datasets show that $H^3$3Fusion is more helpful, less harmful, and more honest from two aspects: it outperforms each individually aligned model by $11.37%$, and it provides stronger robustness compared to the state-of-the-art LLM ensemble approaches by $13.77%$. Code is available at github.com/sftekin/h3fusion.

Problem

• Fusing multiple aligned LLMs to enhance helpful, harmless, and honest capabilities
• Applying mixture-of-experts methodology to dynamically select optimal model components
• Improving alignment robustness and performance beyond individual models and ensembles

Innovation

• Ensembles multiple aligned LLMs for enhanced capabilities
• Uses mixture-of-experts with frozen attention and tuned FFN
• Introduces gating loss and regularization for performance boost