EnsemW2S: Can an Ensemble of LLMs be Leveraged to Obtain a Stronger LLM?

2024-10-06

arXiv.org

AI Summary

This work addresses the challenge of enhancing large language model (LLM) performance in fully unsupervised settings. We propose an “Easy-to-Hard” (E2H) weak-to-strong (W2S) training framework. Methodologically, it introduces an AdaBoost-inspired collaborative supervision mechanism wherein multiple weak LLMs jointly supervise a stronger student LLM via ensemble-based weak supervision—marking the first instance of weak-model ensembles guiding strong-model distillation. The approach integrates ensemble learning, weakly supervised knowledge distillation, curriculum-style difficulty progression, and LLM self-feedback calibration. Evaluated on challenging QA benchmarks, our method achieves average accuracy gains of +5.0% (binary classification) and +4.0% (generation), with peak improvements reaching +14.0%. Several key metrics match those of fully supervised, ground-truth–labeled baselines.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

How can we harness the collective capabilities of multiple Large Language Models (LLMs) to create an even more powerful model? This question forms the foundation of our research, where we propose an innovative approach to weak-to-strong (w2s) generalization-a critical problem in AI alignment. Our work introduces an easy-to-hard (e2h) framework for studying the feasibility of w2s generalization, where weak models trained on simpler tasks collaboratively supervise stronger models on more complex tasks. This setup mirrors real-world challenges, where direct human supervision is limited. To achieve this, we develop a novel AdaBoost-inspired ensemble method, demonstrating that an ensemble of weak supervisors can enhance the performance of stronger LLMs across classification and generative tasks on difficult QA datasets. In several cases, our ensemble approach matches the performance of models trained on ground-truth data, establishing a new benchmark for w2s generalization. We observe an improvement of up to 14% over existing baselines and average improvements of 5% and 4% for binary classification and generative tasks, respectively. This research points to a promising direction for enhancing AI through collective supervision, especially in scenarios where labeled data is sparse or insufficient.

Problem

• Leveraging multiple LLMs to create a stronger model
• Weak-to-strong generalization in AI alignment
• Enhancing LLM performance with ensemble weak supervisors

Innovation

• AdaBoost-inspired ensemble method for LLMs
• Easy-to-hard framework for weak-to-strong generalization
• Collective weak supervision enhances strong LLMs