Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts

2026-06-20

AI Summary

This study addresses the limitations of existing value assessments for large language models, which often rely on single behavioral metrics and fail to capture structural trade-offs within comprehensive moral frameworks. To overcome this, the work introduces Q methodology into LLM value evaluation, establishing a symmetric forced-ranking protocol between humans and models based on 140 moral statements to quantify alignment in value structures. Leveraging Procrustes analysis, RSA-based Spearman correlation, deterministic bucket mapping, and multi-temperature repeated sampling across twelve mainstream models, the research uncovers significant family-level heterogeneity, sensitivity to generative stochasticity, and localized misalignments. These findings reveal that high aggregate scores can mask deep structural discrepancies, thereby advocating for a new evaluation paradigm centered on structural alignment rather than surface-level consistency.

Abstract

Large Language Models (LLMs) are increasingly deployed in contexts requiring complex moral reasoning and value trade-offs. However, existing evaluations typically rely on item-level behavioral metrics, which fail to capture how models structurally prioritize competing values as a cohesive system. To address this, we propose a symmetric human-LLM evaluation framework, grounded in Q methodology, to measure value-structure alignment. Under our protocol, humans and models sort an identical 140-item moral statement set into a shared nine-column forced distribution; for LLMs, we elicit strict rankings and deterministically map them to Q-sort buckets. Using a human reference sample ($N=35$), we establish a stable three-factor reference geometry specific to this instrument and sample. We evaluate 12 LLMs across four model families via 240 replicated Q-sorts at two temperature settings, quantifying structural alignment via Procrustes similarity ($φ$) and RSA-based Spearman correlation ($ρ$). Our results reveal significant cross-family heterogeneity, model-specific sensitivity to generation stochasticity and localized misalignment, which demonstrate that favorable global scores can obscure underlying regional distortions. While rank- and bucket-based analyses remain highly consistent, prompt phrasing introduces notable variance. Ultimately, assessing value-structure alignment provides a crucial structural complement to traditional itemwise moral benchmarks.

Problem

• value-structure alignment
• Large Language Models
• moral reasoning
• Q methodology
• value trade-offs

Innovation

• value-structure alignment
• Q methodology
• symmetric evaluation
• moral reasoning
• Procrustes similarity