PeerBTS: Incentivizing Effort in Strategyproof Peer Selection

2026-05-22

AI Summary

This study addresses the lack of incentives for evaluators to exert effort in existing strategyproof peer selection mechanisms, which often results in poor assessment quality. To remedy this, the paper introduces effort incentives into the framework for the first time and proposes a novel peer prediction lottery mechanism based on the Robust Bayesian Truth Serum (PeerBTS). The mechanism preserves Bayesian-Nash incentive compatibility while effectively motivating truthful effortful evaluations by leveraging aggregate information beyond individual reports. This design aligns incentives with accuracy through collective signals. Non-strategic simulations demonstrate that PeerBTS outperforms current strategyproof methods in both incentivizing effort and improving assessment accuracy, and preliminary empirical evidence further supports the efficacy of the peer prediction approach.

Abstract

Peer selection, the evaluation and selection of agents by their peers, is an important problem in the field of computational social choice; with applications to grading in massively online courses (MOOCs) and academic peer review. Current existing algorithmic and empirical work focuses on developing and analyzing novel \emph{strategyproof} mechanisms, wherein no agent has an incentive to misreport their evaluations. However, the majority of published mechanisms share a flaw: they do not \emph{reward} agents for any effort expended during the evaluation process. In cases where high quality evaluations are costly to produce this missing incentive fails to align agents with an overall goal of accurate selection. To address this gap we first prove theoretically that incentivizing effort in peer selection requires information beyond a single evaluation. We then propose \textsc{PeerBTS}, a mechanism that combines a peer-prediction lottery, leveraging work on the Robust Bayesian Truth Serum, with any existing peer-selection mechanism to incentivize effort while remaining Bayes-Nash incentive compatible. We find that while an incentive-compatible peer-selection mechanism using agent predictions to incentivize effort is possible it requires adjustments to the assumed problem context and limits other mechanistics properties. We additionally present a series of non-strategic simulations to validate incentives and evaluate the performance of PeerBTS relative to existing strategyproof peer selection mechanisms. Finally, we discuss the results of an initial study on the validity of peer-prediction from a small academic workshop.

Problem

• peer selection
• incentive mechanism
• effort elicitation
• strategyproofness
• computational social choice

Innovation

• peer selection
• incentive mechanism
• Bayesian Truth Serum
• effort elicitation
• strategyproofness