Wisdom of the AI Crowd (AI-CROWD) for Ground Truth Approximation in Content Analysis: A Research Protocol&Validation Using Eleven Large Language Models

2026-03-06

AI Summary

This study addresses the challenge of large-scale content analysis, which is often hindered by the absence of ground-truth labels due to the high cost, inefficiency, and inconsistency of manual annotation. To overcome this limitation, the authors propose AI-CROWD, a novel protocol that systematically leverages the collective intelligence of multiple large language models (LLMs) to approximate ground-truth labels. The framework integrates 11 distinct LLMs and employs majority voting to generate consensus labels, while also incorporating diagnostic metrics to identify high-confidence samples and flag potential ambiguities or model biases. Experimental results demonstrate that AI-CROWD can efficiently produce reliable pseudo-labels in the absence of true annotations, offering a scalable, low-cost, and model-agnostic paradigm for large-scale content analysis.

Technology Category

• Humans and AI: Crowd Sourcing and Human Computation
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large-scale content analysis is increasingly limited by the absence of observable ground truth or gold-standard labels, as creating such benchmarks through extensive human coding becomes impractical for massive datasets due to high time, cost, and consistency challenges. To overcome this barrier, we introduce the AI-CROWD protocol, which approximates ground truth by leveraging the collective outputs of an ensemble of large language models (LLMs). Rather than asserting that the resulting labels are true ground truth, the protocol generates a consensus-based approximation derived from convergent and divergent inferences across multiple models. By aggregating outputs via majority voting and interrogating agreement/disagreement patterns with diagnostic metrics, AI-CROWD identifies high-confidence classifications while flagging potential ambiguity or model-specific biases.

Problem

• ground truth approximation
• content analysis
• large language models
• gold-standard labels
• human coding

Innovation

• AI-CROWD
• ground truth approximation
• large language models
• ensemble consensus
• content analysis