RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator

2026-05-20

AI Summary

Existing LLM-as-a-judge benchmarks are largely confined to single-turn question answering, making them inadequate for evaluating the quality of generated text in multi-turn dialogues. This work proposes a synthetic evaluation benchmark tailored for multi-turn conversations, wherein controlled single-turn flaws are injected into reference documents to construct paired dialogues with unambiguous quality labels and precisely localized error turns. To mitigate label noise, the approach dynamically filters samples based on difficulty ratings. Integrating synthetic data generation, a Bradley–Terry ranking model, a random-walk scoring algorithm, and partial observability analysis, the method enables stable ranking of 21 prominent LLM judges across machine learning, biomedical, and financial domains, demonstrating strong cross-domain robustness and evaluation efficacy.

Abstract

As interactive LLM-based applications are created and refined, model developers need to evaluate the quality of generated text along many possible axes. For simpler systems, human evaluation may be practical, but in complicated systems like conversational chatbots, the amount of generated text can overwhelm human annotation resources. Model developers have begun to rely heavily on auto-evaluation, where LLMs are also used to judge generation quality. However, existing LLM-as-a-judge benchmarks largely focus on simple Q\&A tasks that do not match the complexity of multi-turn conversations. We introduce RankJudge, a benchmark generator for evaluating LLM-as-a-judge on multi-turn conversations grounded in reference documents. RankJudge creates pairs of conversations where one conversation has a single flaw injected into one turn. This construction allows paired conversations to be labeled unambiguously as better or worse, and precisely isolates failure categories to individual turns, enabling a strict joint correctness criterion for judging. We implement RankJudge across the domains of machine learning, biomedicine, and finance, evaluate 21 frontier LLM judges, and rank those judges via the Bradley-Terry model. Our formulation also allows ranking each conversation pair with difficulty ratings, which we use to dynamically curate the evaluation slice to reduce label noise, as confirmed via human annotation. We find that judge rankings are stable under partial observability, coarser correctness criteria, and an alternative random-walk rating algorithm.

Problem

• LLM-as-a-judge
• multi-turn conversation
• synthetic benchmark
• auto-evaluation
• conversation quality

Innovation

• LLM-as-a-Judge
• multi-turn conversation
• synthetic benchmark
• paired comparison
• difficulty-aware evaluation