SDGBiasBench: Benchmarking and Mitigating Vision--Language Models' Biases in Sustainable Development Goals

2026-05-20

AI Summary

Existing vision-language models are prone to interference from prior knowledge when evaluating Sustainable Development Goals (SDGs) and lack a comprehensive benchmark that integrates qualitative judgment with quantitative estimation. This work introduces SDGBiasBench, the first large-scale multimodal benchmark for SDG assessment, comprising 500,000 expert-annotated multiple-choice questions and 50,000 regression tasks, which systematically reveals decision-making and estimation biases in SDG reasoning. Furthermore, the authors propose CADE, a training-free, plug-and-play debiasing method that improves multiple-choice accuracy by up to 25% and reduces mean absolute error in regression tasks by as much as 12 points across several mainstream models, substantially mitigating systematic biases in multimodal SDG inference.

Abstract

Assessing progress toward the Sustainable Development Goals (SDGs) requires multi-step reasoning over visual cues, contextual knowledge, and development indicators, where incomplete evidence use and imperfect evidence integration can introduce hidden prediction biases. Real-world SDG monitoring further spans both qualitative judgments and quantitative estimation. However, existing benchmarks typically evaluate these aspects in isolation, obscuring systematic biases that emerge when models substitute priors for evidence. To address this gap, we propose SDGBiasBench, a large-scale benchmark suite for SDG-oriented vision-language reasoning. Spanning 500k expert-involved multiple-choice questions and 50k regression tasks, the benchmark enables comprehensive assessment of both decision-level and estimation-level bias in Vision--Language Models (VLMs). Evaluations on SDGBiasBench reveal an intrinsic SDG bias in current VLMs, where predictions are frequently driven by SDG specific priors rather than reliable multi-modal cues. To mitigate such bias, we propose CADE (Contrastive Adaptive Debias Ensemble), a training-free, plug-and-play method that leverages modality-specific answer priors. CADE yields significant gains on the proposed benchmark, improving multiple-choice accuracy by up to 25% and reducing regression MAE by up to 12 points across multiple VLMs. We hope our work can foster the development of more fair and reliable AI systems for sustainable development.

Problem

• Vision-Language Models
• Sustainable Development Goals
• Bias
• Benchmarking
• Multi-modal Reasoning

Innovation

• SDGBiasBench
• Vision-Language Models
• Bias Mitigation
• CADE
• Sustainable Development Goals