Which Demographics do LLMs Default to During Annotation?

2024-10-11

arXiv.org

AI Summary

This study investigates implicit demographic biases—specifically gender, race, and age preferences—exhibited by large language models (LLMs) in politeness/impoliteness annotation tasks when no explicit demographic information is provided. Method: We introduce a “placebo prompt” controlled paradigm to isolate LLMs’ default annotation behavior, conducting three systematically designed experiments on the POPQUORN dataset: demographic-conditioned prompting, placebo prompting, and zero-shot prompting. Annotations are generated via multi-round LLM inference, followed by rigorous statistical significance testing. Contribution/Results: We demonstrate that LLMs are not demographically neutral in zero-shot settings; instead, they exhibit reproducible, statistically significant demographic biases—most prominently along gender and age dimensions. This work overcomes a key limitation of prior research—namely, the absence of appropriate controls for detecting latent biases—and establishes a novel methodological framework for fairness evaluation in LLM-based annotation systems.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Demographics and cultural background of annotators influence the labels they assign in text annotation -- for instance, an elderly woman might find it offensive to read a message addressed to a"bro", but a male teenager might find it appropriate. It is therefore important to acknowledge label variations to not under-represent members of a society. Two research directions developed out of this observation in the context of using large language models (LLM) for data annotations, namely (1) studying biases and inherent knowledge of LLMs and (2) injecting diversity in the output by manipulating the prompt with demographic information. We combine these two strands of research and ask the question to which demographics an LLM resorts to when no demographics is given. To answer this question, we evaluate which attributes of human annotators LLMs inherently mimic. Furthermore, we compare non-demographic conditioned prompts and placebo-conditioned prompts (e.g.,"you are an annotator who lives in house number 5") to demographics-conditioned prompts ("You are a 45 year old man and an expert on politeness annotation. How do you rate {instance}"). We study these questions for politeness and offensiveness annotations on the POPQUORN data set, a corpus created in a controlled manner to investigate human label variations based on demographics which has not been used for LLM-based analyses so far. We observe notable influences related to gender, race, and age in demographic prompting, which contrasts with previous studies that found no such effects.

Problem

• Investigates default demographics LLMs mimic in annotations
• Compares demographic vs non-demographic prompt effects on labels
• Analyzes bias in politeness/offensiveness annotations across demographics

Innovation

• Studying LLM biases and inherent knowledge
• Injecting diversity via demographic prompts
• Comparing non-demographic and placebo-conditioned prompts