Enhancing Jailbreak Attack Against Large Language Models through Silent Tokens

2024-05-31

arXiv.org

AI Summary

This work uncovers a latent vulnerability in large language models’ (LLMs) alignment mechanisms at the refusal boundary: appending multiple EOS tokens to the end of harmful inputs triggers “context splitting,” causing harmful and harmless inputs to anomalously converge near the refusal boundary in latent space—thereby evading safety filters. To exploit this, we propose BOOST—a lightweight, gradient-free, and human-crafted-prompt-free jailbreaking attack that silently injects EOS tokens at the token level. BOOST significantly reduces LLMs’ perceived risk of harmful content while preserving semantic integrity. Leveraging attention analysis and adversarial prompt engineering, BOOST boosts success rates of four canonical jailbreaking attacks by an average of 32.7% across multiple mainstream LLMs. This is the first systematic demonstration that EOS tokens can be maliciously repurposed, exposing a critical token-level security blind spot in current alignment paradigms.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Along with the remarkable successes of Language language models, recent research also started to explore the security threats of LLMs, including jailbreaking attacks. Attackers carefully craft jailbreaking prompts such that a target LLM will respond to the harmful question. Existing jailbreaking attacks require either human experts or leveraging complicated algorithms to craft jailbreaking prompts. In this paper, we introduce BOOST, a simple attack that leverages only the eos tokens. We demonstrate that rather than constructing complicated jailbreaking prompts, the attacker can simply append a few eos tokens to the end of a harmful question. It will bypass the safety alignment of LLMs and lead to successful jailbreaking attacks. We further apply BOOST to four representative jailbreak methods and show that the attack success rates of these methods can be significantly enhanced by simply adding eos tokens to the prompt. To understand this simple but novel phenomenon, we conduct empirical analyses. Our analysis reveals that adding eos tokens makes the target LLM believe the input is much less harmful, and eos tokens have low attention values and do not affect LLM's understanding of the harmful questions, leading the model to actually respond to the questions. Our findings uncover how fragile an LLM is against jailbreak attacks, motivating the development of strong safety alignment approaches.

Problem

• Exposes hidden weakness in aligned LLMs' refusal boundaries
• Demonstrates eos tokens boost jailbreak attack success rates
• Reveals vulnerability in commercial APIs to eos token manipulation

Innovation

• Appending eos tokens boosts jailbreak attacks
• Probing mechanism reveals commercial API vulnerabilities
• New defenses needed for robust refusal boundaries