PapersPlease: A Benchmark for Evaluating Motivational Values of Large Language Models Based on ERG Theory

2025-06-27

AI Summary

This study investigates implicit hierarchical preferences over human needs—formalized via the ERG theory (Existence, Relatedness, Growth)—in large language models’ (LLMs) role-based decision-making, and their entanglement with social identity bias. We introduce a novel immigration adjudication benchmark comprising 3,700 moral dilemmas, wherein LLMs assume the role of immigration officers and render admission decisions grounded in ERG need prioritization. Methodologically, we integrate formal ERG-theoretic value modeling with role-specific prompting, structured narrative generation, and cross-model statistical hypothesis testing. Evaluating six state-of-the-art LLMs, we identify significant and consistent motivational priority biases across models. Crucially, we demonstrate that socially embedded identities induce systematic discrimination: certain models reject marginalized applicants at rates up to 23.6% higher than their non-marginalized counterparts. These findings expose a previously overlooked structural bias in LLM value alignment—namely, the differential weighting of fundamental human needs—and underscore the necessity of need-aware evaluation frameworks in AI ethics and alignment research.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Evaluating the performance and biases of large language models (LLMs) through role-playing scenarios is becoming increasingly common, as LLMs often exhibit biased behaviors in these contexts. Building on this line of research, we introduce PapersPlease, a benchmark consisting of 3,700 moral dilemmas designed to investigate LLMs' decision-making in prioritizing various levels of human needs. In our setup, LLMs act as immigration inspectors deciding whether to approve or deny entry based on the short narratives of people. These narratives are constructed using the Existence, Relatedness, and Growth (ERG) theory, which categorizes human needs into three hierarchical levels. Our analysis of six LLMs reveals statistically significant patterns in decision-making, suggesting that LLMs encode implicit preferences. Additionally, our evaluation of the impact of incorporating social identities into the narratives shows varying responsiveness based on both motivational needs and identity cues, with some models exhibiting higher denial rates for marginalized identities. All data is publicly available at https://github.com/yeonsuuuu28/papers-please.

Problem

• Evaluating LLM biases in role-playing moral dilemmas
• Assessing LLM decision-making using ERG theory frameworks
• Measuring impact of social identities on LLM responses

Innovation

• Benchmark with 3,700 ERG-based moral dilemmas
• LLMs role-play as immigration inspectors
• Analyze biases via social identity integration