Predicting Disagreement with Human Raters in LLM-as-a-Judge Difficulty Assessment without Using Generation-Time Probability Signals

2026-05-12

AI Summary

This work addresses the significant discrepancies often observed between LLM-as-a-Judge and human raters in the automatic difficulty annotation of educational materials, which undermine annotation reliability. The authors propose a novel approach that does not rely on generative probability signals and, for the first time, enables cross-model comparable disagreement prediction. By leveraging embedding models such as ModernBERT to construct an independent semantic space and exploiting the ordinal nature of difficulty levels, the method identifies potentially inconsistent judgments through geometric consistency among rating sets. Evaluated on the CEFR sentence difficulty assessment task, the approach achieves significantly higher AUC in disagreement prediction than probability-based baselines on both GPT-OSS-120B and Qwen3-235B-A22B, thereby substantially improving the efficiency of human review.

Abstract

Automatic generation of educational materials using large language models (LLMs) is becoming increasingly common, but assigning difficulty levels to such materials still requires substantial human effort. LLM-as-a-Judge has therefore attracted attention, yet disagreement with human raters remains a major challenge. We propose a method for predicting which LLM-generated difficulty ratings are likely to disagree with human raters, so that such cases can be sent for re-rating. Unlike prior approaches, our method does not rely on generation-time probability signals, which must be collected during rating generation and are often difficult to compare across LLMs. Instead, exploiting the fact that difficulty is an ordinal scale, we use a separate embedding space, such as ModernBERT, and identify disagreement candidates based on the geometric consistency of the rating set. Experiments on English CEFR-based sentence difficulty assessment with GPT-OSS-120B and Qwen3-235B-A22B showed that the proposed method achieved higher AUC for predicting disagreement with human raters than probability-based baselines.

Problem

• LLM-as-a-Judge
• difficulty assessment
• human disagreement
• educational materials
• ordinal rating

Innovation

• LLM-as-a-Judge
• disagreement prediction
• geometric consistency
• ordinal difficulty scale
• embedding space