Evaluating Large Language Models on Historical Health Crisis Knowledge in Resource-Limited Settings: A Hybrid Multi-Metric Study

2026-03-20

AI Summary

This study evaluates the reliability and applicability of large language models (LLMs) in disseminating historical public health crisis knowledge in resource-constrained settings. Focusing on COVID-19, dengue, Nipah virus, and chikungunya, the authors construct a multilingual question-answering dataset and introduce a novel hybrid evaluation framework that integrates semantic similarity, natural language inference, and expert–model cross-validation. The framework is systematically applied to assess mainstream LLMs—including GPT-4, Gemini Pro, Llama-3, and Mistral-7B—in low-resource contexts such as Bangladesh. The findings reveal critical limitations and potential risks in the models’ ability to accurately convey complex epidemiological knowledge, offering empirical evidence and cautionary insights for deploying LLMs to support public health decision-making in underserved regions.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large Language Models (LLMs) offer significant potential for delivering health information. However, their reliability in low-resource contexts remains uncertain. This study evaluates GPT-4, Gemini Pro, Llama~3, and Mistral-7B on health crisis-related enquiries concerning COVID-19, dengue, the Nipah virus, and Chikungunya in the low-resource context of Bangladesh. We constructed a question--answer dataset from authoritative sources and assessed model outputs through semantic similarity, expert-model cross-evaluation, and Natural Language Inference (NLI). Findings highlight both the strengths and limitations of LLMs in representing epidemiological history and health crisis knowledge, underscoring their promise and risks for informing policy in resource-constrained environments.

Problem

• Large Language Models
• Health Crisis Knowledge
• Resource-Limited Settings
• Historical Epidemiology
• Model Reliability

Innovation

• hybrid multi-metric evaluation
• large language models
• health crisis knowledge
• resource-limited settings
• natural language inference