Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA

2026-06-26

AI Summary

This study challenges the foundational assumption in the “LLM-as-a-Judge” paradigm that evaluation is inherently easier than generation. By having large language models self-evaluate their own generated answers in a controlled question-answering setting—thereby eliminating external knowledge confounds—the authors systematically compare the intrinsic difficulty of generation versus evaluation tasks. Empirical results across four benchmarks (SQuAD 2.0, DROP, HotpotQA, and MuSiQue) reveal that model accuracy in generation significantly exceeds self-evaluation accuracy on three datasets. Attention analysis further shows that during evaluation, models exhibit substantially weaker focus on both the context and the answer compared to generation. Moreover, LoRA fine-tuning experiments demonstrate negative transfer between the two tasks, providing strong evidence for an inherent asymmetry in their underlying mechanisms.

Abstract

LLM-as-a-Judge and self-evaluation pipelines implicitly assume that evaluation is easier than generation. We test this in a controlled in-context QA setting where a context passage is the sole information source and each model judges the answer it generated, removing the parametric-knowledge confound of open-domain comparisons. Across four benchmarks (SQuAD 2.0, DROP, HotpotQA, MuSiQue) and two models, evaluation is not uniformly easier: generation accuracy exceeds self-evaluation on three of four, with multi-hop MuSiQue the exception. Attention analysis reveals why: evaluation attends to context 3--5x less than generation does and barely reads the candidate answer. LoRA fine-tuning confirms the asymmetry is not a training artifact: generation fine-tuning induces over-acceptance and evaluation fine-tuning degrades generation. These findings challenge core assumptions in self-evaluation pipelines.

Problem

• LLM-as-a-Judge
• self-evaluation
• task asymmetry
• in-context QA
• evaluation vs generation

Innovation

• task asymmetry
• mechanistic interpretability
• in-context QA
• self-evaluation
• LoRA fine-tuning