AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing

2026-06-17

AI Summary

This work addresses the challenge of modeling human preferences in LLM-based auditing systems, which typically rely on high-quality annotations or strong judge outputs—assumptions that often fail in real-world settings where human feedback is sparse and initial judgments are prone to bias. To overcome this limitation, the authors propose AURA, a framework that treats judge reliability as a latent variable dynamically updated with accumulating evidence. AURA employs an uncertainty-aware iterative optimization mechanism to actively select high-uncertainty samples for human review and integrates an evidence propagation algorithm to adaptively enhance judge quality under sparse feedback. Experimental results demonstrate that AURA significantly outperforms existing methods on both synthetic and real-world datasets, effectively improving alignment between LLM judges and human judgments.

Abstract

Large language models (LLMs) are increasingly used as judges for open-ended generation, as large-scale human evaluation is often expensive and difficult to scale, yet their preferences remain imperfect proxies for human judgment. Existing auditing pipelines often assume that a reliable subset of examples or clean supervision signals are available beforehand, for example from human annotation, heuristic filtering, or the outputs of strong judges. In LLM evaluation, this assumption is fragile: the initial split may inherit judge bias, while human verification is typically too scarce to define stable groups at scale. We propose AURA, an adaptive uncertainty--aware refinement framework for auditing pairwise LLM--as--a--judge decisions under selected human verification. AURA iteratively learns a human-consistency signal, propagates reliable evidence, and prioritizes uncertain comparisons for human review. The key idea is to treat trust in a judge as a latent quantity that is progressively refined as evidence accumulates. We provide a compact formulation, a stable refinement procedure, and a comprehensive evaluation on both synthetic and real pairwise LLM-answer data.

Problem

• LLM-as-a-Judge
• auditing
• human consistency
• uncertainty
• bias

Innovation

• Adaptive Uncertainty-aware Refinement
• LLM-as-a-Judge
• Human-consistency Signal
• Iterative Auditing
• Pairwise Evaluation