When One LLM Drools, Multi-LLM Collaboration Rules

2025-02-06

AI Summary

In real-world scenarios, single large language models (LLMs) suffer from data distribution bias, skill heterogeneity, and insufficient demographic representativeness, undermining output reliability for complex, contextualized, and subjective tasks. To address this, we propose a multi-LLM collaboration paradigm and introduce the first systematic framework spanning four interoperability levels: API invocation, textual interaction, logit integration, and weight sharing. This framework explicitly models inter-model information exchange and capability complementarity as essential mechanisms for enhancing reliability, democratic fairness, and pluralistic representation. By unifying multi-granularity techniques—including adaptive orchestration, iterative dialogue, ensemble-based logit fusion, and parameter-space cooperation—it significantly improves task robustness, group-level fairness, and representational inclusivity. Empirical evaluation demonstrates consistent superiority over individual LLMs across reliability metrics and multidimensional representation benchmarks, establishing both theoretical foundations and a scalable methodology for collaborative AI systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-AI Collaboration / Human-AI Teaming

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

This position paper argues that in many realistic (i.e., complex, contextualized, subjective) scenarios, one LLM is not enough to produce a reliable output. We challenge the status quo of relying solely on a single general-purpose LLM and argue for multi-LLM collaboration to better represent the extensive diversity of data, skills, and people. We first posit that a single LLM underrepresents real-world data distributions, heterogeneous skills, and pluralistic populations, and that such representation gaps cannot be trivially patched by further training a single LLM. We then organize existing multi-LLM collaboration methods into a hierarchy, based on the level of access and information exchange, ranging from API-level, text-level, logit-level, to weight-level collaboration. Based on these methods, we highlight how multi-LLM collaboration addresses challenges that a single LLM struggles with, such as reliability, democratization, and pluralism. Finally, we identify the limitations of existing multi-LLM methods and motivate future work. We envision multi-LLM collaboration as an essential path toward compositional intelligence and collaborative AI development.

Problem

• Single LLM insufficient for complex scenarios
• Multi-LLM collaboration enhances data representation
• Addresses reliability, democratization, and pluralism issues

Innovation

• Multi-LLM collaboration enhances reliability
• Hierarchical methods improve data representation
• Collaborative AI fosters compositional intelligence