Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations

2026-01-30

AI Summary

This work addresses the observed self-preference of large language models (LLMs) during evaluation, which may stem from confounding factors such as "narcissism" or task difficulty. To disentangle these influences, the authors propose an evaluator-quality baseline that compares a model’s preference for its own erroneous outputs against those generated by others. This approach reduces measurement error by 89.6% and, when validated across 37,448 queries, reveals that only 51% of the originally reported self-preference effects remain statistically significant under the new baseline. By integrating an automated evaluation framework, significance testing, and entropy analysis, this study substantially corrects prior misattributions of LLM "narcissism" and establishes a more reliable paradigm for evaluating self-preference in language models.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Evaluation and Analysis
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent research has shown that large language models (LLMs) favor their own outputs when acting as judges, undermining the integrity of automated post-training and evaluation workflows. However, it is difficult to disentangle which evaluation biases are explained by narcissism versus general experimental confounds, distorting measurements of self-preference bias. We discover a core methodological confound which could reduce measurement error by 89.6%. Specifically, LLM evaluators may deliver self-preferring verdicts when the judge responds to queries which they completed incorrectly themselves; this would be true regardless of whether one of their responses is their own. To decouple self-preference signals from noisy outputs on hard problems, we introduce an Evaluator Quality Baseline, which compares the probability that a judge incorrectly votes for itself against the probability that it votes for an incorrect response from another model. Evaluating this simple baseline on 37,448 queries, only 51% of initial findings retain statistical significance. Finally, we turn towards characterizing the entropy of"easy"versus"hard"evaluation votes from LLM judges. Our corrective baseline enables future research on self-preference by eliminating noisy data from potential solutions. More widely, this work contributes to the growing body of work on cataloging and isolating judge-bias effects.

Problem

• self-preference
• LLM evaluators
• evaluation bias
• narcissism
• judge bias

Innovation

• self-preference bias
• LLM evaluation
• evaluator quality baseline
• judge bias
• measurement confound