Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation

2026-05-07

AI Summary

This study addresses the lack of systematic understanding regarding how annotator characteristics and textual linguistic properties jointly influence annotation variability in harmful language detection. Integrating sociolinguistic features of annotators—including demographic attributes and attitudinal measures—with computational linguistic metrics of text, the authors conduct large-scale statistical modeling and joint analyses across four harmful language datasets. They uncover significant interaction effects between annotator and text features, demonstrating that lexical cues and annotator attitudes exert strong, interdependent influences on labeling outcomes. Notably, these interaction patterns vary substantially across datasets. These findings challenge prevailing practices that overlook such complexities and underscore the necessity of explicitly accounting for annotator–text interactions when developing and generalizing harmful language detection models.

Abstract

Human label variation has been established as a central phenomenon in NLP: the perspectives different annotators have on the same item need to be embraced. Data collection practices thus shifted towards increasing the annotator numbers and releasing disaggregated datasets, harmful language being most resourced due to its high subjectivity. While this resulted in rich information about \textit{who} annotated (sociodemographics, attitudes, etc.), the \textit{what} (e.g., linguistic properties of items), and their interplay has received little attention. We present the first large-scale analysis of four reference datasets for harmful language detection, bringing together annotator characteristics, linguistic properties of the items, and their interactions in a statistically informed picture. We find that interactions are crucial, revealing intersectional effects ignored in previous work, and that a strong role is played by lexical cues and annotator attitudes. Effect patterns, however, vary considerably across datasets. This urges caution about generalization and transferability.

Problem

• annotation variation
• harmful language
• linguistic features
• annotator characteristics
• subjectivity

Innovation

• annotation variation
• linguistic features
• annotator characteristics
• harmful language detection
• intersectional effects