Annotator Positionality as Signal: Psychometric Weighting for Anti-Autistic Ableism Detection

2026-05-25

AI Summary

This study addresses the limitations of large language models in accurately identifying anti-ableist language related to autism in high-stakes contexts, noting that prevailing evaluation methods largely neglect the perspectives of autistic individuals and their allies. The authors propose a psychometrically grounded, annotator-positionality-weighted evaluation framework that, for the first time, treats annotator identity as a core signal in bias detection. Integrating community-engaged research, context-sensitive analysis, and misattribution diagnostics, this approach constructs a fairness-aware weighted benchmark that surpasses simple majority voting. Their findings reveal that mainstream models frequently misclassify community-reclaimed language as ableist, overrely on keyword matching while disregarding speaker identity and contextual nuance, and exhibit systematic negative biases against autistic communities in masked evaluations—highlighting the critical importance of centering marginalized voices in language model assessment.

Abstract

Large language models (LLMs) are increasingly used in decision-making tasks where they can amplify or suppress perspectives, raising concerns in high-stakes settings affecting autistic communities. While previous research has identified disability-related biases in LLMs, it remains unclear how they conceptualize ableism or detect it in text. We introduce a bias-aware evaluation framework targeting anti-autistic ableist language with a psychometrically-weighted, community-proximate ground truth anchored in annotator positionality. This framework constitutes a stricter standard than conventional majority-vote aggregation which significantly and consistently underweights autistic and autism-accepting perspectives. We find that LLMs frequently produce harmful outputs, mislabel community-reclaimed language as ableist, and express more negative attitudes toward autistic people when assessment instruments are masked. Our error analysis reveals that models rely on surface-level keyword matching rather than contextual factors such as speaker identity, and whether the language fosters in-group solidarity or inflicts out-group harm.

Problem

• anti-autistic ableism
• large language models
• bias detection
• annotator positionality
• disability bias

Innovation

• annotator positionality
• psychometric weighting
• anti-autistic ableism
• bias-aware evaluation
• community-proximate ground truth