Auditing Disability Representation in Vision-Language Models

2026-01-24

AI Summary

This study addresses the tendency of vision-language models to generate outputs that lack visual grounding and reinforce deficit-oriented narratives when describing disability-related images, thereby compromising explanatory fidelity. To tackle this issue, the authors construct the first benchmark dataset comprising paired neutral and disability-context prompts, enabling a systematic evaluation of 15 mainstream models under zero-shot settings. They propose an evaluation framework centered on explanatory fidelity, integrating affective and sociocultural metrics, and introduce an LLM-as-judge protocol validated by annotators with lived disability experience. The findings reveal amplification of bias at the intersection of disability, race, and gender, and demonstrate that targeted prompt engineering and preference-based fine-tuning significantly enhance model fidelity while mitigating harmful narratives.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Humans and AI: AI for Accessibility
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Vision-language models (VLMs) are increasingly deployed in socially sensitive applications, yet their behavior with respect to disability remains underexplored. We study disability aware descriptions for person centric images, where models often transition from evidence grounded factual description to interpretation shift including introduction of unsupported inferences beyond observable visual evidence. To systematically analyze this phenomenon, we introduce a benchmark based on paired Neutral Prompts (NP) and Disability-Contextualised Prompts (DP) and evaluate 15 state-of-the-art open- and closed-source VLMs under a zero-shot setting across 9 disability categories. Our evaluation framework treats interpretive fidelity as core objective and combines standard text-based metrics capturing affective degradation through shifts in sentiment, social regard and response length with an LLM-as-judge protocol, validated by annotators with lived experience of disability. We find that introducing disability context consistently degrades interpretive fidelity, inducing interpretation shifts characterised by speculative inference, narrative elaboration, affective degradation and deficit oriented framing. These effects are further amplified along race and gender dimension. Finally, we demonstrate targeted prompting and preference fine-tuning effectively improves interpretive fidelity and reduces substantially interpretation shifts.

Problem

• disability representation
• vision-language models
• interpretive fidelity
• speculative inference
• affective degradation

Innovation

• interpretive fidelity
• disability representation
• vision-language models
• LLM-as-judge
• prompt engineering