Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks

2026-03-22

AI Summary

This work addresses the challenge that large language models (LLMs) often fail to fairly represent diverse population perspectives in subjective text annotation tasks, largely due to the flawed assumption of a single ground truth. To overcome this limitation, the authors propose a perspective-driven reasoning framework that explicitly models annotation distributions across multiple demographic or social groups, thereby abandoning the notion of a unique true label. Furthermore, they introduce an adaptive human annotation sampling strategy that prioritizes groups for which the LLM performs poorly, given a fixed annotation budget. Evaluated on politeness and offensiveness scoring tasks, the proposed approach significantly improves annotation quality for underrepresented or hard-to-model groups while maintaining broad overall population coverage, outperforming uniform sampling baselines.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Preferences

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models are increasingly used to annotate texts, but their outputs reflect some human perspectives better than others. Existing methods for correcting LLM annotation error assume a single ground truth. However, this assumption fails in subjective tasks where disagreement across demographic groups is meaningful. Here we introduce Perspective-Driven Inference, a method that treats the distribution of annotations across groups as the quantity of interest, and estimates it using a small human annotation budget. We contribute an adaptive sampling strategy that concentrates human annotation effort on groups where LLM proxies are least accurate. We evaluate on politeness and offensiveness rating tasks, showing targeted improvements for harder-to-model demographic groups relative to uniform sampling baselines, while maintaining coverage.

Problem

• subjective tasks
• LLM annotation bias
• demographic disagreement
• annotation fairness
• multi-perspective evaluation

Innovation

• Perspective-Driven Inference
• LLM annotation bias
• adaptive sampling
• subjective tasks
• demographic fairness