EigenBench: A Comparative Behavioral Measure of Value Alignment

2025-09-02

AI Summary

Existing AI value alignment evaluation lacks unsupervised, quantifiable, and objective benchmarks. Method: We propose EigenBench—a black-box benchmark for comparative value alignment assessment of language models—by innovatively adapting the decentralized trust algorithm EigenTrust to alignment evaluation. EigenBench enables relative alignment measurement in domains without ground-truth answers (e.g., ethical judgment) via inter-model peer evaluation and score aggregation, eliminating reliance on labeled reference data. It integrates prompted persona-based behavioral elicitation, model ensembling, and EigenTrust’s consensus mechanism to construct a behavior-driven, unsupervised evaluation framework. Results: Experiments demonstrate that EigenBench effectively discriminates intrinsic value preferences across models, exhibits robustness to prompt perturbations, and sensitively captures self-preference disparities among models—validating its discriminative power and sensitivity.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Aligning AI with human values is a pressing unsolved problem. To address the lack of quantitative metrics for value alignment, we propose EigenBench: a black-box method for comparatively benchmarking language models' values. Given an ensemble of models, a constitution describing a value system, and a dataset of scenarios, our method returns a vector of scores quantifying each model's alignment to the given constitution. To produce these scores, each model judges the outputs of other models across many scenarios, and these judgments are aggregated with EigenTrust (Kamvar et al, 2003), yielding scores that reflect a weighted-average judgment of the whole ensemble. EigenBench uses no ground truth labels, as it is designed to quantify traits for which reasonable judges may disagree on the correct label. Using prompted personas, we test whether EigenBench scores are more sensitive to the model or the prompt: we find that most of the variance is explained by the prompt, but a small residual quantifies the disposition of the model itself.

Problem

• Quantitatively measuring AI value alignment with humans
• Benchmarking language models' values without ground truth
• Assessing model versus prompt influence on alignment

Innovation

• Black-box benchmarking method for models
• EigenTrust algorithm aggregates ensemble judgments
• No ground truth labels required