Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs

2025-08-22

AI Summary

Existing LLM safety evaluations suffer from fundamental ambiguity regarding whether jailbreak success reflects genuine internalization of harmful knowledge. Method: We propose a novel decoupling framework for jailbreak techniques and construct knowledge-intensive QA tasks to systematically assess models across three dimensions: mastery of dangerous knowledge, capability to plan harmful tasks, and robustness in harm judgment. Our methodology integrates dangerous-knowledge probing, judgment-pattern analysis, and a multi-dimensional threat assessment protocol. Contribution/Results: We uncover, for the first time, a severe hallucination loop in the dominant LLM-as-a-judge paradigm: jailbreak success rates are significantly misaligned with actual possession of dangerous knowledge. Experiments demonstrate that current benchmarks overestimate real-world abuse risk; most jailbreaks rely on superficial linguistic pattern matching rather than deep knowledge activation. This work challenges the validity of prevailing safety evaluations and provides theoretical foundations and methodological tools for developing robust, threat-informed assessment frameworks.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Security and Privacy: Large-scale security measurements

Abstract

With the development of Large Language Models (LLMs), numerous efforts have revealed their vulnerabilities to jailbreak attacks. Although these studies have driven the progress in LLMs' safety alignment, it remains unclear whether LLMs have internalized authentic knowledge to deal with real-world crimes, or are merely forced to simulate toxic language patterns. This ambiguity raises concerns that jailbreak success is often attributable to a hallucination loop between jailbroken LLM and judger LLM. By decoupling the use of jailbreak techniques, we construct knowledge-intensive Q&A to investigate the misuse threats of LLMs in terms of dangerous knowledge possession, harmful task planning utility, and harmfulness judgment robustness. Experiments reveal a mismatch between jailbreak success rates and harmful knowledge possession in LLMs, and existing LLM-as-a-judge frameworks tend to anchor harmfulness judgments on toxic language patterns. Our study reveals a gap between existing LLM safety assessments and real-world threat potential.

Problem

• Assessing LLMs' real-world misuse threat beyond jailbreak success rates
• Investigating dangerous knowledge possession versus toxic language simulation
• Evaluating harmfulness judgment robustness in LLM-as-a-judge frameworks

Innovation

• Constructs knowledge-intensive Q&A for threat investigation
• Decouples jailbreak techniques to assess misuse risks
• Evaluates harmful knowledge possession and judgment robustness