BIT.UA-AAUBS at ArchEHR-QA 2026: Evaluating Open-Source and Proprietary LLMs via Prompting in Low-Resource QA

2026-05-05

AI Summary

This study addresses the challenge of clinical question answering and evidence retrieval under stringent medical privacy constraints and in the absence of task-specific training data. The authors propose a prompt engineering framework that integrates task decomposition, chain-of-thought reasoning, and in-context learning, enhanced by a majority voting ensemble and an LLM-as-a-judge mechanism to improve robustness. Evaluations on the ArchEHR-QA 2026 benchmark—using both open-source medical-adapted models such as MedGemma-3-27B and leading closed-source large language models—demonstrate that the proposed zero-shot prompting approach achieves first place in Subtask 4 and third place in Subtask 3 without any fine-tuning, thereby validating the effectiveness and competitiveness of pure prompting strategies in real-world clinical settings.

Abstract

This paper presents the joint participation of the BIT.UA and AAUBS groups in the ArchEHR-QA 2026 shared task, which focuses on clinical question answering and evidence grounding in a low-resource setting. Due to the absence of training data and the strict data privacy constraints inherent to the healthcare domain (e.g. GDPR), we investigate the capabilities of Large Language Models (LLMs) without weight updates. We evaluate several state-of-the-art proprietary models and locally deployable open-source alternatives using various prompt engineering strategies, including task decomposition, Chain-of-Thought, and in-context learning. Furthermore, we explore majority voting and LLM-as-a-judge ensembling techniques to maximize predictive robustness. Our results demonstrate that while proprietary models exhibit strong resilience to prompt variations, domain-adapted open-source models (such as MedGemma 3 27B) achieve highly competitive performance when paired with the right prompt. Overall, our prompt-based approach proved highly effective, securing 1st place in Subtask 4 (evidence citation alignment) and 3rd place in Subtask 3 (patient-friendly answer generation). All code, results, and prompts are available on our GitHub repository: https://github.com/bioinformatics-ua/ArchEHR-QA-2026.

Problem

• low-resource QA
• clinical question answering
• evidence grounding
• data privacy
• Large Language Models

Innovation

• prompt engineering
• low-resource QA
• LLM ensembling
• clinical question answering
• privacy-preserving AI