Smart But Not Moral? Moral Alignment In Human-AI Decision-Making

2026-04-15

AI Summary

This study addresses ethical challenges in high-stakes human-AI collaborative decision-making, where AI systems often provoke concerns regarding fairness, accountability, and harm assessment due to misalignment with human moral intuitions. Moving beyond conventional approaches focused on functional or behavioral alignment, this work innovatively positions moral alignment as a core dimension of human-AI decision integration. It proposes a novel analytical framework grounded in Moral Foundations Theory to evaluate value consistency from multiple stakeholder perspectives. By systematically assessing the congruence between AI decision logic and human moral intuitions, the research demonstrates how moral misalignment adversely impacts AI deployment in high-risk contexts. The findings offer both theoretical grounding and practical guidance for designing AI systems with enhanced moral sensitivity.

Technology Category

• Philosophy and Ethics of AI: Morality & Value-based AI
• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

In high-stakes AI-supported decisions, considerations are not purely technical but involve moral judgments about fairness, responsibility, and harm. While prior research has focused mainly on functional or behavioral alignment, this paper argues that moral alignment may be a more fundamental dimension of human-AI decision-making. Moral alignment is defined as the perceived congruence between the values embedded in an AI system's decision logic and the moral intuitions of stakeholders. Building on Moral Foundations Theory, the paper adopts a multi-stakeholder perspective and highlights why moral (mis)alignment matters for the meaningful integration of AI in sensitive contexts.

Problem

• moral alignment
• human-AI decision-making
• Moral Foundations Theory
• stakeholder values
• AI ethics

Innovation

• moral alignment
• Moral Foundations Theory
• human-AI decision-making
• multi-stakeholder perspective
• value congruence