Collaborative QA using Interacting LLMs. Impact of Network Structure, Node Capability and Distributed Data

2025-11-17

AI Summary

This paper addresses the problem of hallucination propagation in multi-LLM collaborative question answering (CQA), where spurious outputs diffuse through interaction networks and impede accurate answer inference. Methodologically, we propose the first generative framework integrating mean-field dynamics with a stochastic utility model to formally characterize the evolution of ground-truth states within LLM networks. We derive sufficient conditions for the existence and uniqueness of fixed points governing state convergence. Specifically, we employ mean-field approximation to model information diffusion and define dynamic transition probabilities via the stochastic utility model, enabling rigorous convergence analysis on directed heterogeneous networks. Experiments systematically evaluate our framework across 100 open-source LLMs, assessing impacts of data heterogeneity, node capability, network topology, and prompt sensitivity. Results demonstrate significant improvements in suppressing hallucination propagation and enhancing collective reasoning accuracy.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Stochastic Optimization
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In this paper, we model and analyze how a network of interacting LLMs performs collaborative question-answering (CQA) in order to estimate a ground truth given a distributed set of documents. This problem is interesting because LLMs often hallucinate when direct evidence to answer a question is lacking, and these effects become more pronounced in a network of interacting LLMs. The hallucination spreads, causing previously accurate LLMs to hallucinate. We study interacting LLMs and their hallucination by combining novel ideas of mean-field dynamics (MFD) from network science and the randomized utility model from economics to construct a useful generative model. We model the LLM with a latent state that indicates if it is truthful or not with respect to the ground truth, and extend a tractable analytical model considering an MFD to model the diffusion of information in a directed network of LLMs. To specify the probabilities that govern the dynamics of the MFD, we propose a randomized utility model. For a network of LLMs, where each LLM has two possible latent states, we posit sufficient conditions for the existence and uniqueness of a fixed point and analyze the behavior of the fixed point in terms of the incentive (e.g., test-time compute) given to individual LLMs. We experimentally study and analyze the behavior of a network of $100$ open-source LLMs with respect to data heterogeneity, node capability, network structure, and sensitivity to framing on multiple semi-synthetic datasets.

Problem

• Modeling collaborative question-answering in LLM networks
• Analyzing hallucination propagation through network interactions
• Studying fixed-point behavior under varying incentives and structures

Innovation

• Mean-field dynamics model for LLM network interactions
• Randomized utility model for probabilistic dynamics specification
• Latent state tracking for truthfulness in collaborative QA