Decentralized Aggregation of LLM Predictions via Wagering Mechanisms

2026-07-05

AI Summary

This work addresses the challenge of robustly aggregating predictions in decentralized settings without accessing private information from large language models while defending against strategic reporting. The authors propose WALLA, a mechanism that integrates a wagering framework wherein each model simultaneously submits its prediction and a learned wager, with the wager serving as the aggregation weight. By designing a payment rule based on a leave-one-out baseline, WALLA achieves incentive compatibility and aligns wagers with model competence. Notably, it is the first mechanism to guarantee dominant-strategy incentive compatibility, align wagers with model ability, and permit independent optimization of wagering strategies without requiring optimal predictions. Experiments demonstrate that WALLA matches the performance of centralized approaches on question-answering and forecasting tasks under heterogeneous models and private information, while exhibiting uncertainty awareness, incentive compatibility, and decentralized learning capabilities.

Abstract

It is increasingly common to aggregate predictions from multiple LLMs, each with domain expertise or access to private tools and data, to improve collective prediction performance. In decentralized settings, aggregation weights need to be determined without access to models' private information and should remain robust to strategic reporting. We propose a family of advantage-aligned wagering mechanisms for LLM aggregation (WALLA), in which each model reports a prediction and a learned wager, and predictions are aggregated using wagers as weights. WALLA introduces a leave-one-out baseline into the net payout function, yielding three desirable properties: (1) dominant-strategy incentive compatibility of prediction under arbitrary belief structure, (2) advantage--wager alignment, where the optimal wager is proportional to the model's expected score advantage, and (3) prediction-agnostic wager optimization, enabling decentralized learning of wager policies without requiring optimal predictions. We further instantiate two mechanism variants that trade off normality and no-arbitrage while maintaining a bounded worst-case deficit for the mechanism. Experiments on question-answering and forecasting benchmarks across heterogeneous models and private-information settings show that WALLA matches centralized aggregation methods in predictive performance, while simultaneously achieving decentralized learning, advantage-aligned aggregation weights, uncertainty awareness, and incentive-compatible prediction.

Problem

• decentralized aggregation
• LLM predictions
• strategic reporting
• private information
• incentive compatibility

Innovation

• wagering mechanisms
• LLM aggregation
• incentive compatibility
• decentralized learning
• advantage-aligned weighting