Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching

2025-05-27

AI Summary

This paper investigates fundamental limitations on aligning large language models (LLMs) with human preferences within a game-theoretic framework—specifically, whether pairwise human preference data alone can yield a reward function satisfying desirable calibration properties, such as Condorcet consistency, Smith consistency, and strategic diversity. Method: We first derive necessary and sufficient conditions for Smith consistency; then, integrating social choice theory, the Bradley–Terry–Luce model, and zero-sum game modeling, we rigorously prove that perfect preference matching is impossible under the assumption of smooth learnable mappings. Contributions: (1) the first formal criterion for verifying Smith consistency; (2) identification of an inherent theoretical ceiling on LLM alignment—demonstrating that no algorithm relying solely on pairwise preferences can achieve universal calibration; and (3) a robustness criterion establishing an insurmountable fundamental boundary for alignment method design.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Learning Human Values and Preferences
• Game Theory and Economic Paradigms: Social Choice / Voting

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Nash Learning from Human Feedback is a game-theoretic framework for aligning large language models (LLMs) with human preferences by modeling learning as a two-player zero-sum game. However, using raw preference as the payoff in the game highly limits the potential of the game-theoretic LLM alignment framework. In this paper, we systematically study using what choices of payoff based on the pairwise human preferences can yield desirable alignment properties. We establish necessary and sufficient conditions for Condorcet consistency, diversity through mixed strategies, and Smith consistency. These results provide a theoretical foundation for the robustness of game-theoretic LLM alignment. Further, we show the impossibility of preference matching -- i.e., no smooth and learnable mappings of pairwise preferences can guarantee a unique Nash equilibrium that matches a target policy, even under standard assumptions like the Bradley-Terry-Luce model. This result highlights the fundamental limitation of game-theoretic LLM alignment.

Problem

• Study payoff choices for desirable LLM alignment properties
• Establish conditions for Condorcet and Smith consistency
• Prove impossibility of preference matching in Nash equilibrium

Innovation

• Game-theoretic framework for LLM alignment
• Conditions for desirable alignment properties
• Impossibility of preference matching