JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models

2024-06-13

arXiv.org

AI Summary

Current LLM jailbreaking evaluation lacks standardized benchmarks, leading to fragmented methodologies, poor human-AI alignment, and inconsistent cost profiles—severely hindering comparability of attack and defense strategies. To address this, we introduce JailbreakEval, the first systematic evaluation toolkit for jailbreaking. Our approach comprises three key contributions: (1) a comprehensive methodology taxonomy covering 90+ studies; (2) a modular, customizable multi-dimensional evaluation framework integrating five complementary evaluators—rule-based matching, LLM-as-judge, toxicity classification, semantic similarity, and human calibration; and (3) streamlined single-command invocation and workflow orchestration, drastically lowering evaluation barriers and computational overhead. Empirical validation demonstrates robustness and discriminative power across mainstream LLMs. JailbreakEval has been widely adopted by the research community and is fostering consensus on rigorous, reproducible jailbreaking evaluation standards.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Jailbreak attacks induce Large Language Models (LLMs) to generate harmful responses, posing severe misuse threats. Though research on jailbreak attacks and defenses is emerging, there is no consensus on evaluating jailbreaks, i.e., the methods to assess the harmfulness of an LLM's response are varied. Each approach has its own set of strengths and weaknesses, impacting their alignment with human values, as well as the time and financial cost. This diversity challenges researchers in choosing suitable evaluation methods and comparing different attacks and defenses. In this paper, we conduct a comprehensive analysis of jailbreak evaluation methodologies, drawing from nearly 90 jailbreak research published between May 2023 and April 2024. Our study introduces a systematic taxonomy of jailbreak evaluators, offering indepth insights into their strengths and weaknesses, along with the current status of their adaptation. To aid further research, we propose JailbreakEval, a toolkit for evaluating jailbreak attempts. JailbreakEval includes various evaluators out-of-the-box, enabling users to obtain results with a single command or customized evaluation workflows. In summary, we regard JailbreakEval to be a catalyst that simplifies the evaluation process in jailbreak research and fosters an inclusive standard for jailbreak evaluation within the community.

Problem

• Evaluating jailbreak attempts in LLMs
• Assessing harmfulness of LLM responses
• Standardizing jailbreak evaluation methodologies

Innovation

• Integrated toolkit for jailbreak evaluation
• Systematic taxonomy of evaluators
• Customizable evaluation workflows