Deviation Ratings: A General, Clone-Invariant Rating Method

2025-02-17

AI Summary

In N-player non-zero-sum multi-agent/multi-task evaluation, strategy redundancy—e.g., model cloning—induces biased scoring, undermining fairness and robustness. Method: We propose the first clone-invariant bias-aware scoring method, grounded in coarse correlated equilibria (CCE) to model strategy values. This extends clone invariance from two-player zero-sum games to general N-player games for the first time, unifying adversarial, cooperative, and mixed-motive interactions. Leveraging strategy-space redundancy analysis and an LLM-based multidimensional evaluation framework, we validate our method across multiple benchmarks—including large language model evaluation suites. Results: Our bias score significantly mitigates interference from semantically similar strategies on pairwise evaluations, enhancing both robustness and fairness. The core contribution is the first clone-invariant scoring paradigm for general-sum games, providing both theoretical foundations and practical tools for trustworthy multi-agent system evaluation.

Technology Category

• Game Theory and Economic Paradigms: Fair Division
• Multiagent Systems: Adversarial Agents
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Many real-world multi-agent or multi-task evaluation scenarios can be naturally modelled as normal-form games due to inherent strategic (adversarial, cooperative, and mixed motive) interactions. These strategic interactions may be agentic (e.g. players trying to win), fundamental (e.g. cost vs quality), or complementary (e.g. niche finding and specialization). In such a formulation, it is the strategies (actions, policies, agents, models, tasks, prompts, etc.) that are rated. However, the rating problem is complicated by redundancy and complexity of N-player strategic interactions. Repeated or similar strategies can distort ratings for those that counter or complement them. Previous work proposed ``clone invariant'' ratings to handle such redundancies, but this was limited to two-player zero-sum (i.e. strictly competitive) interactions. This work introduces the first N-player general-sum clone invariant rating, called deviation ratings, based on coarse correlated equilibria. The rating is explored on several domains including LLMs evaluation.

Problem

• N-player general-sum clone invariant rating
• strategic interactions in multi-agent scenarios
• rating distortion due to strategy redundancy

Innovation

• N-player general-sum clone invariant rating
• Based on coarse correlated equilibria
• Applied in multi-agent evaluation scenarios