Value Compass Leaderboard: A Platform for Fundamental and Validated Evaluation of LLMs Values

2025-01-13

AI Summary

Current LLM value alignment evaluation faces three fundamental challenges: (1) insufficient capacity to uncover models’ latent value orientations, (2) low validity of static assessment methods, and (3) neglect of cultural and individual value diversity. To address these, we introduce the first comprehensive benchmark platform for value alignment. Our approach innovates in three ways: (1) a motivation-based foundational value theory framework for systematic modeling of LLMs’ underlying values; (2) a generative, dynamic evaluation paradigm that synthesizes behavior-driven test cases to enhance ecological validity and temporal relevance; and (3) a weighted, multidimensional alignment metric supporting cross-cultural and inter-individual value heterogeneity. Empirical results demonstrate substantial improvements in assessment validity, generalizability, and interpretability across diverse LLMs and value contexts.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Morality & Value-based AI

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

As Large Language Models (LLMs) achieve remarkable breakthroughs, aligning their values with humans has become imperative for their responsible development and customized applications. However, there still lack evaluations of LLMs values that fulfill three desirable goals. (1) Value Clarification: We expect to clarify the underlying values of LLMs precisely and comprehensively, while current evaluations focus narrowly on safety risks such as bias and toxicity. (2) Evaluation Validity: Existing static, open-source benchmarks are prone to data contamination and quickly become obsolete as LLMs evolve. Additionally, these discriminative evaluations uncover LLMs' knowledge about values, rather than valid assessments of LLMs' behavioral conformity to values. (3) Value Pluralism: The pluralistic nature of human values across individuals and cultures is largely ignored in measuring LLMs value alignment. To address these challenges, we presents the Value Compass Leaderboard, with three correspondingly designed modules. It (i) grounds the evaluation on motivationally distinct extit{basic values to clarify LLMs' underlying values from a holistic view; (ii) applies a extit{generative evolving evaluation framework with adaptive test items for evolving LLMs and direct value recognition from behaviors in realistic scenarios; (iii) propose a metric that quantifies LLMs alignment with a specific value as a weighted sum over multiple dimensions, with weights determined by pluralistic values.

Problem

• Value Alignment
• Large Language Models
• Cultural Diversity

Innovation

• Value Alignment
• Dynamic Testing
• Comprehensive Scoring System