SocioEval: A Template-Based Framework for Evaluating Socioeconomic Status Bias in Foundation Models

2026-04-02

AI Summary

This study addresses the lack of a systematic evaluation framework for socioeconomic status (SES) bias in large language models (LLMs) during high-stakes decision-making. The authors formally define and quantify this bias by constructing a template-based benchmark encompassing 8 themes, 18 topics, and 240 prompts. They analyze 3,120 model responses across 13 mainstream LLMs under six class-comparison configurations through a three-stage human annotation process. Results reveal a substantial variation in bias prevalence (0.42%–33.75%), with lifestyle-related judgments exhibiting bias levels ten times higher than those in educational decisions. Furthermore, while existing mitigation strategies partially alleviate explicit discrimination, they prove ineffective against implicit stereotypes. This work establishes a scalable, fine-grained benchmark for auditing class-based biases in LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

As Large Language Models (LLMs) increasingly power decision-making systems across critical domains, understanding and mitigating their biases becomes essential for responsible AI deployment. Although bias assessment frameworks have proliferated for attributes such as race and gender, socioeconomic status bias remains significantly underexplored despite its widespread implications in the real world. We introduce SocioEval, a template-based framework for systematically evaluating socioeconomic bias in foundation models through decision-making tasks. Our hierarchical framework encompasses 8 themes and 18 topics, generating 240 prompts across 6 class-pair combinations. We evaluated 13 frontier LLMs on 3,120 responses using a rigorous three-stage annotation protocol, revealing substantial variation in bias rates (0.42\%-33.75\%). Our findings demonstrate that bias manifests differently across themes lifestyle judgments show 10$\times$ higher bias than education-related decisions and that deployment safeguards effectively prevent explicit discrimination but show brittleness to domain-specific stereotypes. SocioEval provides a scalable, extensible foundation for auditing class-based bias in language models.

Problem

• socioeconomic status bias
• foundation models
• bias evaluation
• large language models
• algorithmic fairness

Innovation

• socioeconomic bias
• template-based evaluation
• foundation models
• bias auditing
• LLM fairness