XChoice: Explainable Evaluation of AI-Human Alignment in LLM-based Constrained Choice Decision Making

2026-01-16

AI Summary

This work addresses a critical limitation in current evaluation methods for large language models (LLMs), which predominantly focus on outcome consistency—such as accuracy—and fail to uncover deeper alignment issues regarding how humans and models weigh decision factors under constrained choices. To bridge this gap, the authors propose XChoice, a novel mechanism-driven, interpretable evaluation framework that fits mechanistic decision models to both human and model behaviors, recovering interpretable parameters that quantify alignment in terms of factor weighting, sensitivity to constraints, and implicit trade-offs. Experiments on the ATUS dataset reveal significant alignment disparities for Black and married individuals, while also demonstrating that retrieval-augmented generation (RAG) interventions can effectively mitigate certain biases, thereby validating XChoice’s utility in diagnosing heterogeneous alignment patterns and guiding targeted model improvements.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We present XChoice, an explainable framework for evaluating AI-human alignment in constrained decision making. Moving beyond outcome agreement such as accuracy and F1 score, XChoice fits a mechanism-based decision model to human data and LLM-generated decisions, recovering interpretable parameters that capture the relative importance of decision factors, constraint sensitivity, and implied trade-offs. Alignment is assessed by comparing these parameter vectors across models, options, and subgroups. We demonstrate XChoice on Americans'daily time allocation using the American Time Use Survey (ATUS) as human ground truth, revealing heterogeneous alignment across models and activities and salient misalignment concentrated in Black and married groups. We further validate robustness of XChoice via an invariance analysis and evaluate targeted mitigation with a retrieval augmented generation (RAG) intervention. Overall, XChoice provides mechanism-based metrics that diagnose misalignment and support informed improvements beyond surface outcome matching.

Problem

• AI-human alignment
• constrained choice
• explainable evaluation
• decision making
• large language models

Innovation

• explainable AI
• AI-human alignment
• mechanism-based modeling
• constrained decision making
• retrieval-augmented generation