Structured Disagreement in Health-Literacy Annotation: Epistemic Stability, Conceptual Difficulty, and Agreement-Stratified Inference

2026-04-21

AI Summary

This study challenges the conventional NLP assumption of a single ground-truth label by examining structured disagreement among annotators, particularly in interpretive tasks such as health literacy assessment. Analyzing proportional correctness ratings from multiple annotators across 6,323 open-ended responses about COVID-19 from South America, the authors model the full judgment distribution rather than aggregated labels. Their analysis reveals that task “conceptual difficulty”—not individual annotator variability—is the primary source of disagreement. Through variance decomposition and consistency-stratified inference, they demonstrate that sociodemographic factors such as country, education level, and urban–rural residence exhibit not only magnitude shifts but also directional reversals in their effects across different levels of annotation consistency. These findings underscore the necessity of strong perspectival modeling to enhance the validity of social scientific inference.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Application Domains: Humanities & Computational Social Science
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web

Abstract

Annotation pipelines in Natural Language Processing (NLP) commonly assume a single latent ground truth per instance and resolve disagreement through label aggregation. Perspectivist approaches challenge this view by treating disagreement as potentially informative rather than erroneous. We present a large-scale analysis of graded health-literacy annotations from 6,323 open-ended COVID-19 responses collected in Ecuador and Peru. Each response was independently labeled by multiple annotators using proportional correctness scores, reflecting the degree to which responses align with normative public-health guidelines, allowing us to analyze the full distribution of judgments rather than aggregated labels. Variance decomposition shows that question-level conceptual difficulty accounts for substantially more variance than annotator identity, indicating that disagreement is structured by the task itself rather than driven by individual raters. Agreement-stratified analyses further reveal that key social-scientific effects, including country, education, and urban-rural differences, vary in magnitude and in some cases reverse direction across levels of inter-annotator agreement. These findings suggest that graded health-literacy evaluation contains both epistemically stable and unstable components, and that aggregating across them can obscure important inferential differences. We therefore argue that strong perspectivist modeling is not only conceptually justified but statistically necessary for valid inference in graded interpretive tasks.

Problem

• health-literacy annotation
• structured disagreement
• epistemic stability
• conceptual difficulty
• agreement-stratified inference

Innovation

• perspectivist annotation
• graded health-literacy
• agreement-stratified inference
• conceptual difficulty
• epistemic stability