CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs

2025-05-16

AI Summary

Existing safety evaluation benchmarks for large language models (LLMs) in healthcare lack clinical specificity, fine-grained harm categorization, and comprehensive coverage of jailbreaking attacks. Method: We propose CARES—the first safety and adversarial robustness benchmark tailored to medical LLMs—comprising 18,000+ clinically grounded prompts, structured along eight safety principles, four harm severity levels, and four prompt styles. It introduces a novel ternary response protocol (Accept/Caution/Refuse) and a quantitative Safety Score. Our methodology integrates multi-level human annotation, a lightweight jailbreak detector, and reminder-based conditional response regulation. Contribution/Results: Systematic evaluation across 20+ state-of-the-art medical LLMs reveals that role-playing and steganographic prompts reduce refusal rates by up to 47%; our approach improves safety response accuracy by 32%, substantially mitigating both over-refusal and under-refusal failures.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Security and Privacy: Large-scale security measurements

Abstract

Large language models (LLMs) are increasingly deployed in medical contexts, raising critical concerns about safety, alignment, and susceptibility to adversarial manipulation. While prior benchmarks assess model refusal capabilities for harmful prompts, they often lack clinical specificity, graded harmfulness levels, and coverage of jailbreak-style attacks. We introduce CARES (Clinical Adversarial Robustness and Evaluation of Safety), a benchmark for evaluating LLM safety in healthcare. CARES includes over 18,000 prompts spanning eight medical safety principles, four harm levels, and four prompting styles: direct, indirect, obfuscated, and role-play, to simulate both malicious and benign use cases. We propose a three-way response evaluation protocol (Accept, Caution, Refuse) and a fine-grained Safety Score metric to assess model behavior. Our analysis reveals that many state-of-the-art LLMs remain vulnerable to jailbreaks that subtly rephrase harmful prompts, while also over-refusing safe but atypically phrased queries. Finally, we propose a mitigation strategy using a lightweight classifier to detect jailbreak attempts and steer models toward safer behavior via reminder-based conditioning. CARES provides a rigorous framework for testing and improving medical LLM safety under adversarial and ambiguous conditions.

Problem

• Evaluates safety and adversarial robustness in medical LLMs
• Addresses lack of clinical specificity in existing benchmarks
• Mitigates jailbreak vulnerabilities and over-refusal in medical queries

Innovation

• Introduces CARES benchmark for medical LLM safety
• Proposes three-way response evaluation protocol
• Uses lightweight classifier for jailbreak detection