How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking

2026-05-18

AI Summary

This study addresses the absence of benchmark datasets for Bengali medical visual question answering (MedVQA) by introducing BanglaMedVQA—the first low-resource MedVQA dataset comprising clinically validated image-question-answer triplets in Bengali. Leveraging this dataset, the authors conduct a systematic evaluation of prominent large language models and vision-language models, including GPT-4o mini, Gemini, and Gemma-3. Experimental results reveal that current models exhibit substantially lower performance on Bengali MedVQA compared to English benchmarks, with particularly poor accuracy on complex diagnostic questions. These findings highlight a critical gap in the capability of existing multimodal models to perform fine-grained medical reasoning in low-resource languages. This work establishes essential infrastructure and empirical evidence for advancing multilingual evaluation frameworks in medical artificial intelligence.

Abstract

Recent advancements in Large Language Models (LLMs) and Large Vision Language Models (LVLMs) have enabled general-purpose systems to demonstrate promising capabilities in complex reasoning tasks, including those in the medical domain. Medical Visual Question Answering (MedVQA) has particularly benefited from these developments. However, despite Bangla being one of the most widely spoken languages globally, there exists no established MedVQA benchmark for it. To address this gap, we introduce BanglaMedVQA, a dataset comprising clinically validated image-question-answer pairs, along with a comprehensive evaluation of current foundation models on this resource. Consistent with prior findings that report low performance of current models on English MedVQA benchmarks, our analysis reveals that Bangla performance is substantially lower, reflecting the challenges inherent to low-resource languages. Even top-performing models such as Gemini and GPT-4.1 mini fail to accurately answer specialized diagnostic questions, indicating severe limitations in fine-grained medical reasoning. Although certain open-source models, such as Gemma-3, occasionally outperform these models in general categories, they too struggle with clinically complex questions, underscoring the urgent need for top-notch evaluation method.

Problem

• Medical Visual Question Answering
• Bangla
• Low-resource languages
• Benchmarking
• Dataset

Innovation

• BanglaMedVQA
• Medical Visual Question Answering
• Low-resource Languages
• Multimodal Benchmarking
• Clinical Reasoning