Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale

2026-01-06

arXiv.org

AI Summary

This study investigates how rating scale design influences the alignment between large language models (LLMs) and human raters when LLMs serve as evaluators, with particular attention to how scale choice moderates human–LLM agreement across different task types. Human and LLM ratings were collected using three common rating scales on six benchmark tasks spanning objective, subjective, and mixed categories, and absolute agreement was quantified via intraclass correlation coefficients (ICC). The work reveals, for the first time, that a 0–5 point scale achieves the highest cross-task human–LLM alignment. It further demonstrates that scale selection significantly affects alignment magnitude and that aggregate metrics may obscure substantial heterogeneity across tasks. Additionally, systematic alignment disparities are identified within gender subgroups.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) are increasingly used as automated evaluators, yet prior works demonstrate that these LLM judges often lack consistency in scoring when the prompt is altered. However, the effect of the grading scale itself remains underexplored. We study the LLM-as-a-judge problem by comparing two kinds of raters: humans and LLMs. We collect ratings from both groups on three scales and across six benchmarks that include objective, open-ended subjective, and mixed tasks. Using intraclass correlation coefficients (ICC) to measure absolute agreement, we find that LLM judgments are not perfectly consistent across scales on subjective benchmarks, and that the choice of scale substantially shifts human-LLM agreement, even when within-group panel reliability is high. Aggregated over tasks, the grading scale of 0-5 yields the strongest human-LLM alignment. We further demonstrate that pooled reliability can mask benchmark heterogeneity and reveal systematic subgroup differences in alignment across gender groups, strengthening the importance of scale design and sub-level diagnostics as essential components of LLM-as-a-judge protocols.

Problem

• LLM-as-a-Judge
• grading scale
• human-LLM alignment
• rating consistency
• subjective evaluation

Innovation

• LLM-as-a-Judge
• grading scale
• human-LLM alignment
• intraclass correlation
• subgroup analysis