Let LLMs Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering

2026-06-13

AI Summary

This work addresses the limitations of large language models in medical question answering—specifically their insufficient accuracy, interpretability, and robustness—by proposing a multi-agent collaborative reasoning framework that incorporates a peer-review mechanism into the inference process for the first time. In this approach, multiple models independently generate chain-of-thought rationales and subsequently evaluate each other’s outputs for factual correctness and logical rigor. The final answer is selected from the highest-scoring reasoning chain, prioritizing reasoning quality over mere answer consistency. Evaluated on HeadQA, MedQA-USMLE, and PubMedQA, the method achieves an average accuracy of 0.820, significantly outperforming both single-model baselines (0.777) and majority voting (up to 0.789), with performance consistently improving as the number of participating models increases.

Abstract

Objective: To enhance the accuracy, interpretability, and robustness of large language models (LLMs) in medical question answering (MedQA). Method: We designed a multi-agent peer-reviewed reasoning method in which multiple LLM agents independently generate chain-of-thought reasoning with candidate answers, then act as peer reviewers to evaluate each other's reasoning for factual correctness and logical soundness. The highest-rated reasoning chain is selected to produce the final answer. Experiments were conducted with five state-of-the-art LLMs (Llama-3.1-8B, Qwen2.5-7B, Phi-4, DeepSeek-LLM-7B, GPT-oss-20B) on three benchmark datasets: HeadQA, MedQA-USMLE, and PubMedQA. Performance was compared against single-model chain-of-thought reasoning and chain-of-thought-based majority voting. Results: Peer-reviewed reasoning consistently outperformed both baselines. The best model combination achieved an average accuracy of 0.820 across datasets, exceeding the strongest single model (0.777) and majority voting ensembles (up to 0.789). The method also scaled effectively with more participating models, while peer assessments reliably distinguished high- from low-quality reasoning chains. Conclusion: The proposed multi-agent peer-reviewed reasoning method enables LLMs to act as both solvers and evaluators, yielding superior performance in MedQA. By emphasizing reasoning quality rather than answer agreement alone, this approach improves accuracy, interpretability, and robustness, offering a promising direction for trustworthy biomedical AI systems.

Problem

• medical question answering
• large language models
• accuracy
• interpretability
• robustness

Innovation

• multi-agent peer review
• chain-of-thought reasoning
• medical question answering
• large language models
• reasoning evaluation