First, do NOHARM: towards clinically safe large language models

2025-11-30

AI Summary

Current large language models (LLMs) lack rigorous clinical safety evaluation in medical consultation, particularly within real-world referral scenarios. Method: We introduce NOHARM, the first clinical harm benchmark grounded in 100 authentic, cross-specialty (10 domains) referral cases, accompanied by 4,249 expert-annotated clinical management options and 12,747 fine-grained annotations—enabling systematic quantification of harm frequency and severity in LLM-generated recommendations. Results: Up to 22.2% of cases entail severe harm risk, with 76.6% attributable to omission errors; the best-performing single LLM surpasses average clinicians in safety by 9.7%; integrating multi-agent collaboration further reduces harm incidence by 8.0%. This work establishes “clinical safety” as a distinct, measurable evaluation dimension, exposes a critical gap between existing benchmarks and real-world clinical risk, and provides a reproducible, scalable assessment framework for the safe deployment of medical LLMs.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Agreement, Argumentation & Negotiation

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Large language models (LLMs) are routinely used by physicians and patients for medical advice, yet their clinical safety profiles remain poorly characterized. We present NOHARM (Numerous Options Harm Assessment for Risk in Medicine), a benchmark using 100 real primary-care-to-specialist consultation cases to measure harm frequency and severity from LLM-generated medical recommendations. NOHARM covers 10 specialties, with 12,747 expert annotations for 4,249 clinical management options. Across 31 LLMs, severe harm occurs in up to 22.2% (95% CI 21.6-22.8%) of cases, with harms of omission accounting for 76.6% (95% CI 76.4-76.8%) of errors. Safety performance is only moderately correlated (r = 0.61-0.64) with existing AI and medical knowledge benchmarks. The best models outperform generalist physicians on safety (mean difference 9.7%, 95% CI 7.0-12.5%), and a diverse multi-agent approach reduces harm compared to solo models (mean difference 8.0%, 95% CI 4.0-12.1%). Therefore, despite strong performance on existing evaluations, widely used AI models can produce severely harmful medical advice at nontrivial rates, underscoring clinical safety as a distinct performance dimension necessitating explicit measurement.

Problem

• Assessing harm frequency and severity in LLM medical advice
• Evaluating clinical safety across multiple medical specialties
• Identifying gaps between AI knowledge benchmarks and actual safety

Innovation

• Developed NOHARM benchmark for clinical safety assessment
• Used multi-agent approach to reduce harmful medical advice
• Measured harm frequency and severity across 31 LLMs