Robust Conformal Consensus: Multi-Agent LLM-as-a-Judge Interval Evaluation with Conformal Prediction

2026-09-06

AI Summary

本文针对多代理LLM评价中的不确定性问题，提出了一种基于保形预测的鲁棒估计框架，通过构建多个LLM评分的预测区间来提高评估结果的稳定性和可靠性。

Abstract

LLM-as-a-Judge has emerged as a promising paradigm for evaluating natural language generation. However, the uncertainty associated with such evaluations remains largely unexplored, which limits their reliability in real-world applications. Although conformal prediction offers a principled framework for uncertainty quantification, existing approaches typically apply it to a single LLM judge, overlooking the variability introduced by using different LLM evaluators. In this work, we propose a robust uncertainty estimation framework for multi-agent LLM-as-a-Judge evaluation. Our approach constructs conformal prediction intervals for LLM-based scores from multiple LLMs. By considering intervals from different LLM judges, we obtain more stable and reliable uncertainty estimates. Extensive experiments demonstrate that our method produces valid prediction intervals with coverage guarantees, and that interval-based aggregation across multiple judges leads to more stable evaluation outcomes.

Problem

• LLM-as-a-Judge
• uncertainty quantification
• conformal prediction
• multi-agent evaluation

Innovation

• Conformal Prediction
• Multi-Agent LLM-as-a-Judge
• Uncertainty Estimation
• Interval Evaluation