AI Alignment From Social Choice Perspectives

2026-06-19

AI Summary

This work addresses the challenge of aligning language models with human feedback when such feedback reflects divergent values, a scenario in which existing alignment methods struggle to effectively aggregate pluralistic perspectives. Drawing on social choice theory, the paper systematically analyzes preference aggregation in alignment processes and identifies failure modes of current approaches when confronted with conflicting feedback. Building on this analysis, the authors propose a principled aggregation framework that expands the design space of alignment algorithms, offering both a theoretical foundation and a novel pathway toward robust and fair language model alignment under value pluralism.

Abstract

Alignment from human feedback uses human judgments about model outputs to steer the behavior of language models after pretraining. When those judgments reflect conflicting views of desirable behavior, the learned objective becomes an aggregate determination of what the model should prefer. We survey recent work that has studied this aggregation problem through the lens of social choice theory. We illustrate how the social choice perspective helps identify failure modes in the feedback aggregation layer and reveals a broader design space for handling disagreement in explicit and principled ways.

Problem

• AI alignment
• human feedback
• social choice theory
• preference aggregation
• disagreement handling

Innovation

• AI alignment
• social choice theory
• preference aggregation
• human feedback
• disagreement handling