Strategic Candidacy in Generative AI Arenas

2026-03-27

AI Summary

In generative AI leaderboards, submitters often manipulate rankings by submitting model clones, undermining their credibility. This work proposes You-Rank-We-Rank (YRWR), a novel mechanism that requires participants to rank their own models and uses these self-rankings to calibrate pairwise preference–based quality estimates. YRWR is the first ranking mechanism provably approximately cloneproof, integrating game-theoretic analysis, statistical correction, and simulation experiments to theoretically demonstrate and empirically validate its effectiveness in deterring strategic cloning. Even when submitters provide partially inaccurate self-rankings, YRWR significantly improves overall ranking accuracy, offering a robust solution to enhance leaderboard integrity in competitive generative AI evaluation settings.

Abstract

AI arenas, which rank generative models from pairwise preferences of users, are a popular method for measuring the relative performance of models in the course of their organic use. Because rankings are computed from noisy preferences, there is a concern that model producers can exploit this randomness by submitting many models (e.g., multiple variants of essentially the same model) and thereby artificially improve the rank of their top models. This can lead to degradations in the quality, and therefore the usefulness, of the ranking. In this paper, we begin by establishing, both theoretically and in simulations calibrated to data from the platform Arena (formerly LMArena, Chatbot Arena), conditions under which producers can benefit from submitting clones when their goal is to be ranked highly. We then propose a new mechanism for ranking models from pairwise comparisons, called You-Rank-We-Rank (YRWR). It requires that producers submit rankings over their own models and uses these rankings to correct statistical estimates of model quality. We prove that this mechanism is approximately clone-robust, in the sense that a producer cannot improve their rank much by doing anything other than submitting each of their unique models exactly once. Moreover, to the extent that model producers are able to correctly rank their own models, YRWR improves overall ranking accuracy. In further simulations, we show that indeed the mechanism is approximately clone-robust and quantify improvements to ranking accuracy, even under producer misranking.

Problem

• strategic candidacy
• clone manipulation
• AI arena
• pairwise preference
• ranking robustness

Innovation

• clone-robust ranking
• You-Rank-We-Rank
• strategic candidacy
• pairwise comparison
• generative AI evaluation