AttackEval: How to Evaluate the Effectiveness of Jailbreak Attacking on Large Language Models

2024-01-17

arXiv.org

AI Summary

To address the lack of systematic and quantitative evaluation for jailbreak attacks against large language models (LLMs), this paper proposes the first dedicated quantitative assessment framework for jailbreak prompts. Methodologically, it moves beyond conventional binary robustness evaluation by introducing a dual-dimensional 0–1 scoring system: coarse-grained (success in bypassing safety alignment) and fine-grained (harm severity, semantic stealthiness, etc.). The framework integrates a multi-model collaborative scoring pipeline, adversarial prompt semantic similarity analysis, and human-in-the-loop verification. Additionally, we release the first high-quality, manually curated jailbreak prompt benchmark, covering diverse attack strategies. Experiments demonstrate that our framework significantly improves detection sensitivity—identifying high-risk jailbreak prompts missed by traditional methods—and substantially enhances LLM safety risk discovery capability.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Ensuring the security of large language models (LLMs) against attacks has become increasingly urgent, with jailbreak attacks representing one of the most sophisticated threats. To deal with such risks, we introduce an innovative framework that can help evaluate the effectiveness of jailbreak attacks on LLMs. Unlike traditional binary evaluations focusing solely on the robustness of LLMs, our method assesses the effectiveness of the attacking prompts themselves. We present two distinct evaluation frameworks: a coarse-grained evaluation and a fine-grained evaluation. Each framework uses a scoring range from 0 to 1, offering unique perspectives and allowing for the assessment of attack effectiveness in different scenarios. Additionally, we develop a comprehensive ground truth dataset specifically tailored for jailbreak prompts. This dataset serves as a crucial benchmark for our current study and provides a foundational resource for future research. By comparing with traditional evaluation methods, our study shows that the current results align with baseline metrics while offering a more nuanced and fine-grained assessment. It also helps identify potentially harmful attack prompts that might appear harmless in traditional evaluations. Overall, our work establishes a solid foundation for assessing a broader range of attack prompts in the area of prompt injection.

Problem

• Evaluate jailbreak attack effectiveness on large language models.
• Develop frameworks for coarse-grained and fine-grained attack assessments.
• Create a ground truth dataset for jailbreak prompt evaluation.

Innovation

• Introduces framework for jailbreak attack evaluation
• Develops ground truth dataset for jailbreak prompts
• Offers coarse and fine-grained evaluation frameworks