The Impact of Annotator Personas on LLM Behavior Across the Perspectivism Spectrum

2025-08-23

AI Summary

This work investigates how large language models (LLMs) model annotator personas across the data perspectivism spectrum—from strong personalization to weak aggregation—to detect hate speech and offensive content. Method: We introduce demographically grounded, predefined annotator personas and integrate them with LLM-generated annotations and established perspective modeling techniques to systematically evaluate annotation consistency and human alignment under varying perspective strengths. Results: LLMs selectively leverage demographic cues in personas to produce annotations biased toward group consensus; they significantly outperform both conventional methods and human annotators under weak-perspective settings, while approaching—though not surpassing—human performance under strong-perspective conditions, revealing a “depersonalized aggregation” tendency. This study is the first to systematically incorporate data perspectivism into LLM-based content annotation, establishing a novel paradigm for interpretable and controllable modeling of subjectivity.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

In this work, we explore the capability of Large Language Models (LLMs) to annotate hate speech and abusiveness while considering predefined annotator personas within the strong-to-weak data perspectivism spectra. We evaluated LLM-generated annotations against existing annotator modeling techniques for perspective modeling. Our findings show that LLMs selectively use demographic attributes from the personas. We identified prototypical annotators, with persona features that show varying degrees of alignment with the original human annotators. Within the data perspectivism paradigm, annotator modeling techniques that do not explicitly rely on annotator information performed better under weak data perspectivism compared to both strong data perspectivism and human annotations, suggesting LLM-generated views tend towards aggregation despite subjective prompting. However, for more personalized datasets tailored to strong perspectivism, the performance of LLM annotator modeling approached, but did not exceed, human annotators.

Problem

• Evaluating LLM annotation of hate speech using personas
• Assessing alignment between LLM-generated and human annotations
• Comparing performance across strong-to-weak perspectivism spectra

Innovation

• LLMs annotate hate speech using personas
• Evaluated against existing perspective modeling techniques
• Performance varies across perspectivism spectra