AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset

2024-11-23

arXiv.org

AI Summary

Large language models (LLMs) lack rigorous evaluation in African healthcare contexts, raising concerns about their clinical reliability, equity, and contextual validity. Method: We introduce AfriMed-QA—the first pan-African, multi-specialty, academic-grade English medical QA benchmark—comprising 15,000 expert-annotated questions spanning 16 countries, 32 specialties, and 60+ medical schools, coupled with a multidimensional evaluation framework assessing correctness, bias, and explainability. Contribution/Results: Our systematic analysis reveals significant performance gaps, geographic biases, and specialty imbalances in LLMs under African conditions: state-of-the-art models underperform substantially on AfriMed-QA relative to USMLE benchmarks; biomedical-specialized models outperform general-purpose LLMs; and lightweight edge models consistently fail basic competency thresholds. Surprisingly, human expert evaluators preferred LLM-generated answers over those provided by practicing clinicians—a finding that challenges conventional clinical QA assessment paradigms. This work establishes a foundational benchmark and methodology for evaluating AI fairness and clinical utility in resource-constrained settings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent advancements in large language model(LLM) performance on medical multiple choice question (MCQ) benchmarks have stimulated interest from healthcare providers and patients globally. Particularly in low-and middle-income countries (LMICs) facing acute physician shortages and lack of specialists, LLMs offer a potentially scalable pathway to enhance healthcare access and reduce costs. However, their effectiveness in the Global South, especially across the African continent, remains to be established. In this work, we introduce AfriMed-QA, the first large scale Pan-African English multi-specialty medical Question-Answering (QA) dataset, 15,000 questions (open and closed-ended) sourced from over 60 medical schools across 16 countries, covering 32 medical specialties. We further evaluate 30 LLMs across multiple axes including correctness and demographic bias. Our findings show significant performance variation across specialties and geographies, MCQ performance clearly lags USMLE (MedQA). We find that biomedical LLMs underperform general models and smaller edge-friendly LLMs struggle to achieve a passing score. Interestingly, human evaluations show a consistent consumer preference for LLM answers and explanations when compared with clinician answers.

Problem

• Large Language Models
• AfriMed-QA
• African Healthcare

Innovation

• AfriMed-QA Dataset
• Medical Knowledge Assessment
• Model Performance Comparison