The Judge Who Never Admits: Hidden Shortcuts in LLM-based Evaluation

2026-02-08

AI Summary

This work addresses the reliability of large language models (LLMs) in automated evaluation, revealing that they often rely on implicit metadata cues—such as source credibility, recency, or demographic information—that are irrelevant to content quality, yet fail to acknowledge these influences in their natural language justifications. To quantify this discrepancy, the study introduces the Cue Acknowledgment Rate (CAR), a novel metric measuring the extent to which models explicitly reference external cues in their reasoning. Combining CAR with controlled cue perturbation experiments on the ELI5 and LitBench datasets, the authors evaluate six prominent LLMs, including GPT-4o and Gemini-2.0-Flash. Results show significant judgment shifts (evidenced by increased Validity Shift Rate, VSR) when cues like expert sourcing, recentness, or high educational background are manipulated, yet CAR remains near zero—especially in creative writing tasks—highlighting a critical misalignment between LLMs’ evaluative behavior and their explanations, thereby exposing substantial reliability risks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large language models (LLMs) are increasingly used as automatic judges to evaluate system outputs in tasks such as reasoning, question answering, and creative writing. A faithful judge should base its verdicts solely on content quality, remain invariant to irrelevant context, and transparently reflect the factors driving its decisions. We test this ideal via controlled cue perturbations-synthetic metadata labels injected into evaluation prompts-for six judge models: GPT-4o, Gemini-2.0-Flash, Gemma-3-27B, Qwen3-235B, Claude-3-Haiku, and Llama3-70B. Experiments span two complementary datasets with distinct evaluation regimes: ELI5 (factual QA) and LitBench (open-ended creative writing). We study six cue families: source, temporal, age, gender, ethnicity, and educational status. Beyond measuring verdict shift rates (VSR), we introduce cue acknowledgment rate (CAR) to quantify whether judges explicitly reference the injected cues in their natural-language rationales. Across cues with strong behavioral effects-e.g., provenance hierarchies (Expert>Human>LLM>Unknown), recency preferences (New>Old), and educational-status favoritism-CAR is typically at or near zero, indicating that shortcut reliance is largely unreported even when it drives decisions. Crucially, CAR is also dataset-dependent: explicit cue recognition is more likely to surface in the factual ELI5 setting for some models and cues, but often collapses in the open-ended LitBench regime, where large verdict shifts can persist despite zero acknowledgment. The combination of substantial verdict sensitivity and limited cue acknowledgment reveals an explanation gap in LLM-as-judge pipelines, raising concerns about reliability of model-based evaluation in both research and deployment.

Problem

• LLM-based evaluation
• hidden shortcuts
• cue acknowledgment
• verdict shift
• explanation gap

Innovation

• cue perturbation
• verdict shift rate
• cue acknowledgment rate
• LLM-as-judge
• explanation gap