Red Teaming Large Language Models for Healthcare

2025-05-01

AI Summary

This paper addresses the challenges of identifying potential clinical safety vulnerabilities in large language models (LLMs) for healthcare and the lack of standardized evaluation protocols. We propose the first red-teaming framework co-designed and deeply informed by clinical domain experts. Methodologically, it integrates clinical-knowledge-guided adversarial prompting, multi-expert collaborative vulnerability annotation, cross-model consistency validation, and taxonomy-driven analysis. Systematically applied to real-world clinical scenarios, it uncovers and categorizes over ten classes of high-risk medical vulnerabilities, with reproducible validation across multiple mainstream LLMs. Key contributions include: (1) uncovering semantic-level medical vulnerabilities inaccessible to technical teams alone; (2) establishing a reproducible, extensible cross-model vulnerability assessment framework; and (3) releasing the first community-driven benchmark and practical guideline for evaluating medical LLM safety—thereby advancing standardization in healthcare AI safety assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We present the design process and findings of the pre-conference workshop at the Machine Learning for Healthcare Conference (2024) entitled Red Teaming Large Language Models for Healthcare, which took place on August 15, 2024. Conference participants, comprising a mix of computational and clinical expertise, attempted to discover vulnerabilities -- realistic clinical prompts for which a large language model (LLM) outputs a response that could cause clinical harm. Red-teaming with clinicians enables the identification of LLM vulnerabilities that may not be recognised by LLM developers lacking clinical expertise. We report the vulnerabilities found, categorise them, and present the results of a replication study assessing the vulnerabilities across all LLMs provided.

Problem

• Identify vulnerabilities in healthcare LLM responses
• Assess clinical harm risks from LLM outputs
• Evaluate LLM safety gaps with clinician input

Innovation

• Red-teaming LLMs with clinical expertise
• Identifying harmful LLM vulnerabilities in healthcare
• Categorizing and replicating found vulnerabilities