'Rich Dad, Poor Lad': How do Large Language Models Contextualize Socioeconomic Factors in College Admission ?

2025-09-19

AI Summary

This study investigates the sensitivity and latent socioeconomic bias of large language models (LLMs) in high-stakes decision-making contexts—specifically university admissions—where fairness and equity are critical. Method: Drawing on cognitive science’s dual-process theory, we propose a Dual-Process Auditing Framework (DPAF) and construct a synthetic dataset of 30,000 applicant profiles. We conduct 5 million prompt-based experiments across four open-source LLMs under both intuitive (System 1) and deliberative (System 2) reasoning modes. Contribution/Results: We identify a systematic low-socioeconomic-status (SES) preference: LLMs consistently favor applicants from lower-income backgrounds—even when academic qualifications are equivalent—and this bias intensifies under System 2 reasoning requiring deeper inference. This is the first work to uncover and characterize such implicit preference mechanisms and their plasticity in fairness-sensitive AI decisions, establishing an auditable, attributable evaluation paradigm for AI governance.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLMs) are increasingly involved in high-stakes domains, yet how they reason about socially sensitive decisions remains underexplored. We present a large-scale audit of LLMs' treatment of socioeconomic status (SES) in college admissions decisions using a novel dual-process framework inspired by cognitive science. Leveraging a synthetic dataset of 30,000 applicant profiles grounded in real-world correlations, we prompt 4 open-source LLMs (Qwen 2, Mistral v0.3, Gemma 2, Llama 3.1) under 2 modes: a fast, decision-only setup (System 1) and a slower, explanation-based setup (System 2). Results from 5 million prompts reveal that LLMs consistently favor low-SES applicants -- even when controlling for academic performance -- and that System 2 amplifies this tendency by explicitly invoking SES as compensatory justification, highlighting both their potential and volatility as decision-makers. We then propose DPAF, a dual-process audit framework to probe LLMs' reasoning behaviors in sensitive applications.

Problem

• Investigating how LLMs incorporate socioeconomic factors in college admissions
• Auditing LLMs' reasoning about sensitive social decisions using dual-process framework
• Analyzing SES bias amplification in LLM decision-making across different prompting modes

Innovation

• Uses dual-process framework from cognitive science
• Leverages synthetic dataset with 30,000 profiles
• Proposes DPAF framework for auditing LLM reasoning