Medical Reasoning in LLMs: An In-Depth Analysis of DeepSeek R1

2025-03-27

AI Summary

This paper addresses the insufficient reliability of large language models (e.g., DeepSeek R1) in clinical diagnostic reasoning. To this end, we propose the first multidimensional evaluation framework tailored to diagnostic decision-making, systematically analyzing model–expert consistency across 100 MedQA cases. Methodologically, we integrate error attribution analysis, response length statistics, and clinical guideline adherence assessment. Our analysis reveals, for the first time, a significant negative correlation between reasoning length and diagnostic accuracy—models achieve higher accuracy when responses remain under 5,000 characters—and identifies six attributable error patterns, including anchoring bias and failure to resolve conflicting information. Experimental results demonstrate a 93% diagnostic accuracy rate. We further derive seven structured improvement directions, establish an interpretability threshold, and delineate a principled reasoning optimization pathway—thereby providing both theoretical foundations and practical paradigms for the safe deployment of medical LLMs.

Technology Category

• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Integrating large language models (LLMs) like DeepSeek R1 into healthcare requires rigorous evaluation of their reasoning alignment with clinical expertise. This study assesses DeepSeek R1's medical reasoning against expert patterns using 100 MedQA clinical cases. The model achieved 93% diagnostic accuracy, demonstrating systematic clinical judgment through differential diagnosis, guideline-based treatment selection, and integration of patient-specific factors. However, error analysis of seven incorrect cases revealed persistent limitations: anchoring bias, challenges reconciling conflicting data, insufficient exploration of alternatives, overthinking, knowledge gaps, and premature prioritization of definitive treatment over intermediate care. Crucially, reasoning length correlated with accuracy - shorter responses (<5,000 characters) were more reliable, suggesting extended explanations may signal uncertainty or rationalization of errors. While DeepSeek R1 exhibits foundational clinical reasoning capabilities, recurring flaws highlight critical areas for refinement, including bias mitigation, knowledge updates, and structured reasoning frameworks. These findings underscore LLMs' potential to augment medical decision-making through artificial reasoning but emphasize the need for domain-specific validation, interpretability safeguards, and confidence metrics (e.g., response length thresholds) to ensure reliability in real-world applications.

Problem

• Evaluating DeepSeek R1's medical reasoning alignment with clinical expertise
• Identifying persistent limitations in LLM-based clinical decision-making
• Improving reliability through bias mitigation and structured reasoning frameworks

Innovation

• DeepSeek R1 achieves 93% diagnostic accuracy
• Uses differential diagnosis and guideline-based treatment
• Identifies bias mitigation and knowledge updates