Metritocracy: Representative Metrics for Lite Benchmarks

2025-06-11

AI Summary

Existing LLM evaluation lacks theoretically grounded methods for selecting representative subsets of metrics. Method: This work pioneers the application of social choice theory to metric representativeness modeling, formally defining two verifiable fairness axioms—*position representativeness* and *position proportionality*—and proposing the first social-choice-based metric selection framework. It derives tight theoretical bounds on subset size via combinatorial optimization and theoretical computer science techniques, and extends the framework to support grouped constraints and generalized representativeness optimization. Results: The framework achieves provable guarantees while enabling efficient, lightweight benchmark design. Empirical validation on real-world LLM benchmarks and hospital quality assessment datasets confirms its effectiveness and practical utility. This work establishes a novel paradigm and rigorous theoretical foundation for verifiable, resource-efficient evaluation benchmark construction.

Technology Category

• Game Theory and Economic Paradigms: Social Choice / Voting
• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

A common problem in LLM evaluation is how to choose a subset of metrics from a full suite of possible metrics. Subset selection is usually done for efficiency or interpretability reasons, and the goal is often to select a ``representative'' subset of metrics. However, ``representative'' is rarely clearly defined. In this work, we use ideas from social choice theory to formalize two notions of representation for the selection of a subset of evaluation metrics. We first introduce positional representation, which guarantees every alternative is sufficiently represented at every position cutoff. We then introduce positional proportionality, which guarantees no alternative is proportionally over- or under-represented by more than a small error at any position. We prove upper and lower bounds on the smallest number of metrics needed to guarantee either of these properties in the worst case. We also study a generalized form of each property that allows for additional input on groups of metrics that must be represented. Finally, we tie theory to practice through real-world case studies on both LLM evaluation and hospital quality evaluation.

Problem

• Formalizing representative subset selection for LLM evaluation metrics
• Introducing positional representation and proportionality in metric selection
• Applying social choice theory to real-world evaluation case studies

Innovation

• Formalizes representation using social choice theory
• Introduces positional representation and proportionality
• Applies theory to LLM and hospital evaluations