Can LLMs Rank? A Tale of Triads and Triage

2026-06-29

AI Summary

This study addresses the reliability of large language models (LLMs) in high-stakes resource allocation ranking tasks by proposing a pairwise comparison–based, two-dimensional evaluation framework. Integrating cycle triplet analysis from social choice theory with rank distance metrics, the approach uniquely unifies the classical consistency coefficient ζ and Kendall’s τ to jointly quantify internal consistency and cross-run stability in LLM-generated rankings. Empirical evaluations on two real-world scenarios—homeless housing allocation and emergency triage—reveal significant disparities in consistency performance across mainstream LLMs, thereby validating the efficacy of the proposed framework and offering practitioners an actionable methodology for assessing LLM-based ranking systems.

Abstract

From housing allocation for households experiencing homelessness to triage in emergency departments, LLMs are increasingly being considered as judges of consequential decisions that require ranking people for scarce resources. Ranking large groups simultaneously is cognitively demanding and error-prone. A natural solution, drawing on decades of social choice theory, elicits pairwise comparisons and aggregates them into a total order. However, a fundamental question remains when LLMs serve as the pairwise judge: how can a practitioner tell, before committing to a ranking, whether the LLM's judgments are sufficiently consistent to trust the result? We discuss two different ways of identifying consistency. A classical diagnostic, the coefficient of consistency $ζ$, originally developed to measure judge reliability by counting circular triads in tournament graphs, provides a cheap, model-free measure of intra-run consistency. Various standard measures of distance between rankings, for example Kendall's $τ$, can measure inter-run variability. We show, in both theory and practice, that these measures are independently valuable, and advocate for using both to assess reliability of rankings. We demonstrate the practical importance of our results across two high-stakes prioritization tasks: homelessness service allocation and emergency department triage. Three different leading LLMs have considerably different performance profiles across these two axes of consistency. We provide guidelines for how practitioners could think about measuring and assessing consistency before committing to a model for ranking or prioritization.

Problem

• LLM ranking
• consistency evaluation
• pairwise comparison
• resource allocation
• social choice

Innovation

• consistency evaluation
• pairwise comparison
• large language models
• social choice theory
• ranking reliability