LLM Judges Can Be Too Generous When There Is No Reference Answer

2026-07-14

AI Summary

This work addresses the unreliability of large language models (LLMs) as open-ended response evaluators in reference-free settings, where they systematically overestimate incorrect answers. To mitigate this bias, the authors propose a two-stage calibration framework: first assessing the evaluator model’s task-specific knowledge, then systematically probing how the presence and placement of reference answers in prompts influence its judgments. Through multilingual controlled experiments, prompt engineering, human annotation comparisons, and sensitivity analyses, the study reveals that LLMs exhibit substantially elevated error rates when evaluating without references; however, incorporating reference answers triggers judgment reversals in up to 85% of cases, yielding evaluations markedly more aligned with human judgments. This approach establishes a general and reliable calibration paradigm for reference-free LLM-based assessment.

Abstract

LLM judges are increasingly being used to evaluate open-ended model responses, often in no-reference settings where a ground-truth answer is unavailable. However, can they reliably assess in such evaluation setups? We explore this question in this paper through a two stage pipeline with a) calibration experiments that assess the judge model's knowledge of the task it is evaluating, and b) sensitivity experiments that assess how the judge model's performance is impacted by the presence and positioning of the reference answer in the prompt. Across experiments covering three languages, we show that the judge models we evaluated tend to over-credit incorrect answers in the absence of a reference answer, and adding reference answer information to the prompt flips the judge model's correct/incorrect decisions by as much as 85% in some experimental settings. Comparison with a subset of human annotations shows that these reference-driven changes generally align with human judgments. Our results emphasize the need for calibrating the LLM judges with a sample with reference-aware evaluation before using them in reference-free setups reliably, and our methodology provides a blueprint for researchers and practitioners in doing such calibration of LLM judges for other tasks.

Problem

• LLM judges
• reference-free evaluation
• evaluation reliability
• open-ended responses
• answer assessment

Innovation

• LLM judges
• reference-free evaluation
• calibration
• sensitivity analysis
• human alignment