Sample Complexity of Peer Prediction

2026-08-17

AI Summary

This study addresses the sample complexity challenge in unbiased mutual information estimation for peer prediction. It identifies the unique estimable mutual information structure under small-sample regimes and proposes an improved dMI estimator alongside a score-based adaptive sampling method. Overcoming traditional estimation bottlenecks, this approach achieves efficient unbiased estimation with an expected sample size below three, significantly reducing variance and optimizing convergence. These contributions provide both theoretical foundations and effective algorithms for reliable mutual information measurement in low-data scenarios, substantially enhancing sample efficiency.

Abstract

Peer prediction seeks to incentivize agents to truthfully report an observed signal by rewarding joint sets of reports without observing a ground truth. Following the generalization of information-theoretic mutual information introduced in Kong and Schoenebeck (2019), we call a function of a joint distribution over signals a mutual information when it is non-negative and disincentivizes garbling reports for all information structures. An unbiased estimator for a mutual information takes some number of samples from the distribution and returns rewards for both agents, such that the expected reward is equal to the mutual information. We seek to characterize the set of mutual informations with unbiased estimators for a given number of samples. We show that for three or fewer sampled report pairs, the only mutual information with an unbiased estimator is trivially zero, and for four or five samples with a binary report space, the Determinant Mutual Information (DMI) of Kong (2024) is the unique mutual information (up to a scalar multiple). We further show that DMI ceases to be unique at six samples. We provide an improved estimator of DMI for any given number of samples and characterize its convergence rate. We also examine mutual information estimators that accept a randomized number of samples. First, we show that mutual information estimators on an ex-ante bounded number of samples (termed "stop-short estimators") can achieve a lower variance than an equivalent fixed-sample estimator (for DMI). Second, we introduce the class of scoring-rule-based mutual informations and identify in this family a mutual information that can be estimated with under three samples in expectation.

Problem

• Peer Prediction
• Sample Complexity
• Mutual Information
• Unbiased Estimator
• Determinant Mutual Information

Innovation

• Peer Prediction
• Sample Complexity
• Determinant Mutual Information
• Unbiased Estimator
• Scoring Rule