Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMs

2025-06-16

AI Summary

Existing LLM moral evaluation frameworks rely excessively on pre-scripted scenarios, prioritize outcome prediction over reasoning process analysis, and fail to detect informational gaps. Method: We propose the first five-dimensional moral capability assessment framework—comprising moral feature identification, importance weighing, reason attribution, judgment integration, and informational gap detection—formalizing philosophical moral skill theory into quantifiable, multi-dimensional metrics. We introduce a dual-experiment paradigm using controllably noisy ethical vignettes, integrating human annotation, dimension-wise scoring, and cross-group comparison (non-experts, philosophers, LLMs). Contribution/Results: Results show LLMs outperform laypeople in standard scenarios but exhibit significant deficits across multiple dimensions in high-noise, novel scenarios—revealing that current evaluations substantially overestimate their genuine moral reasoning capacity.

Technology Category

• Philosophy and Ethics of AI: Morality & Value-based AI
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Moral competence is the ability to act in accordance with moral principles. As large language models (LLMs) are increasingly deployed in situations demanding moral competence, there is increasing interest in evaluating this ability empirically. We review existing literature and identify three significant shortcoming: (i) Over-reliance on prepackaged moral scenarios with explicitly highlighted moral features; (ii) Focus on verdict prediction rather than moral reasoning; and (iii) Inadequate testing of models' (in)ability to recognize when additional information is needed. Grounded in philosophical research on moral skill, we then introduce a novel method for assessing moral competence in LLMs. Our approach moves beyond simple verdict comparisons to evaluate five dimensions of moral competence: identifying morally relevant features, weighting their importance, assigning moral reasons to these features, synthesizing coherent moral judgments, and recognizing information gaps. We conduct two experiments comparing six leading LLMs against non-expert humans and professional philosophers. In our first experiment using ethical vignettes standard to existing work, LLMs generally outperformed non-expert humans across multiple dimensions of moral reasoning. However, our second experiment, featuring novel scenarios designed to test moral sensitivity by embedding relevant features among irrelevant details, revealed a striking reversal: several LLMs performed significantly worse than humans. Our findings suggest that current evaluations may substantially overestimate LLMs' moral reasoning capabilities by eliminating the task of discerning moral relevance from noisy information, which we take to be a prerequisite for genuine moral skill. This work provides a more nuanced framework for assessing AI moral competence and highlights important directions for improving moral competence in advanced AI systems.

Problem

• Assessing moral competence in LLMs beyond prepackaged scenarios
• Evaluating moral reasoning instead of just verdict prediction
• Testing LLMs' ability to recognize information gaps in moral judgments

Innovation

• Multi-dimensional assessment of moral competence
• Novel scenarios testing moral sensitivity
• Framework comparing LLMs with humans