FAIR_XAI: Improving Multimodal Foundation Model Fairness via Explainability for Wellbeing Assessment

2026-04-26

AI Summary

Multimodal foundation models in mental health assessment often compromise clinical fairness and reliability due to insufficient transparency and demographic biases. This work proposes the first explainable artificial intelligence (XAI)-driven fairness intervention framework, systematically evaluating and optimizing diagnostic consistency and group fairness of vision-language models in depression prediction. Experiments on the AFAR-BSFT and E-DAIC datasets demonstrate that Phi3.5-Vision achieves 80.4% accuracy on E-DAIC, whereas Qwen2-VL attains only 33.9%. Fairness-aware prompting enforces equal opportunity at the cost of performance, while explanation-based interventions enhance procedural consistency but may inadvertently exacerbate racial bias. The study reveals a fundamental tension between procedural transparency and outcome fairness, and introduces a novel paradigm for jointly optimizing accuracy, fairness, and cross-domain generalization.

Technology Category

• Computer Vision: Multi-modal Vision
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

In recent years, the integration of multimodal machine learning in wellbeing assessment has offered transformative potential for monitoring mental health. However, with the rapid advancement of Vision-Language Models (VLMs), their deployment in clinical settings has raised concerns due to their lack of transparency and potential for bias. While previous research has explored the intersection of fairness and Explainable AI (XAI), its application to VLMs for wellbeing assessment and depression prediction remains under-explored. This work investigates VLM performance across laboratory (AFAR-BSFT) and naturalistic (E-DAIC) datasets, focusing on diagnostic reliability and demographic fairness. Performance varied substantially across environments and architectures; Phi3.5-Vision achieved 80.4% accuracy on E-DAIC, while Qwen2-VL struggled at 33.9%. Additionally, both models demonstrated a tendency to over-predict depression on AFAR-BSFT. Although bias existed across both architectures, Qwen2-VL showed higher gender disparities, while Phi-3.5-Vision exhibited more racial bias. Our XAI intervention framework yielded mixed results; fairness prompting achieved perfect equal opportunity for Qwen2-VL at a severe accuracy cost on E-DAIC. On AFAR-BSFT, explainability-based interventions improved procedural consistency but did not guarantee outcome fairness, sometimes amplifying racial bias. These results highlight a persistent gap between procedural transparency and equitable outcomes. We analyse these findings and consolidate concrete recommendations for addressing them, emphasising that future fairness interventions must jointly optimise predictive accuracy, demographic parity, and cross-domain generalisation.

Problem

• fairness
• explainable AI
• vision-language models
• wellbeing assessment
• bias

Innovation

• Explainable AI (XAI)
• Vision-Language Models (VLMs)
• Algorithmic Fairness
• Multimodal Wellbeing Assessment
• Bias Mitigation