🤖 AI Summary
This study addresses the limited reliability of large language models in identifying Schwartz’s ten basic human values, particularly their tendency to confuse adjacent value dimensions. The authors construct a balanced dataset comprising 1,000 Russian-language contextual prompts grounded in Schwartz’s theory and conduct the first systematic analysis of directional confusion patterns across 21 instruction-tuned models. They propose a comprehensive evaluation framework integrating precision, rank recovery, and directional error metrics, validated through human annotation, Top-1/3 accuracy, semantic panel aggregation, and zero-model baselines. Experimental results reveal average Acc@1 and Acc@3 scores of 0.683 and 0.892, respectively, with 50.9% of errors concentrated among neighboring values. The study further identifies eight recurrent, highly asymmetric directional confusions across models, highlighting potential biases that may distort higher-level value profiling.
📝 Abstract
Large language models are increasingly evaluated through the values they endorse, but such evaluations presuppose that models can identify the value expressed in a concrete situation. We study this prerequisite as controlled top-1 recognition over Schwartz's ten basic values. Our evaluation set contains 1,000 Russian situational texts, balanced across the ten values and independently labeled by two human annotators per item. We evaluate 21 instruction-tuned LLM runs under a fixed ranked-response protocol; 20 runs with reliable outputs form the semantic panel. Pooled Acc@1 is 0.683 and Acc@3 is 0.892, showing that models often locate the correct motivational region while ranking close alternatives unstably. Adjacent values account for 50.9% of semantic errors, compared with 24.4% under a checkpoint-specific null. Eight directed confusions recur across checkpoints and human-confirmed subsets. Several are strongly asymmetric, including Universalism to Benevolence, Tradition to Conformity, and Security to Power, whereas Stimulation-Hedonism forms a bidirectional boundary. Their severity is checkpoint-specific and can bias higher-order value profiles. The results motivate value-recognition evaluation that combines exact accuracy, ranked recovery, and directed error analysis.