Leaderboard Incentives: Model Rankings under Strategic Post-Training

2026-03-09

AI Summary

This work addresses a critical flaw in current mainstream benchmarks: their incentive structures encourage developers to overfit to leaderboard rankings—a practice known as “benchmaxxing”—thereby obscuring true model capabilities. For the first time, the authors frame benchmark evaluation as a Stackelberg game between the benchmark designer and multiple developers, enabling a formal game-theoretic analysis of how different evaluation protocols shape developer strategies and resulting rankings. Theoretical analysis reveals that existing protocols generally lack a Nash equilibrium, leading to unstable or misleading rankings. In contrast, the proposed “tune-before-test” mechanism is shown to guarantee a unique Nash equilibrium under mild conditions, ensuring that leaderboard rankings faithfully reflect the underlying quality of models.

Technology Category

• Game Theory and Economic Paradigms: Mechanism Design
• Multiagent Systems: Mechanism Design
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Influential benchmarks incentivize competing model developers to strategically allocate post-training resources toward improvements on the leaderboard, a phenomenon dubbed benchmaxxing or training on the test task. In this work, we initiate a principled study of the incentive structure that benchmarks induce. We model benchmarking as a Stackelberg game between a benchmark designer who chooses an evaluation protocol and multiple model developers who compete simultaneously in a subgame given by the designer's choice. Each competitor has a model of unknown latent quality and can inflate its observed score by allocating resources to benchmark-specific improvements. First, we prove that current benchmarks induce games for which no Nash equilibrium between model developers exists. This result suggests one explanation for why current practice leads to misaligned incentives, prompting model developers to strategize in opaque ways. However, we prove that under mild conditions, a recently proposed evaluation protocol, called tune-before-test, induces a benchmark with a unique Nash equilibrium that ranks models by latent quality. This positive result demonstrates that benchmarks need not set bad incentives, even if current evaluations do.

Problem

• leaderboard incentives
• benchmaxxing
• evaluation protocol
• strategic post-training
• model ranking

Innovation

• Stackelberg game
• benchmark design
• Nash equilibrium
• tune-before-test
• strategic post-training