SafeAligner: Safety Alignment against Jailbreak Attacks via Response Disparity Guidance

2024-06-26

arXiv.org

AI Summary

To address the failure of safety alignment in large language models (LLMs) under jailbreak attacks, this paper proposes a decoding-phase safety enhancement method. The approach employs a dual-model architecture comprising a sentinel model and an intruder model; by contrasting their response-level safety scores, it dynamically reweights and guides the token-level output distribution in real time—without fine-tuning the target LLM. Its core innovation is the first proposed *response-difference guidance mechanism*, which achieves high generalizability, low computational overhead, and strong adaptability across diverse attack types. Experiments demonstrate that the method achieves over 92% defense success rate across multiple benchmarks, significantly increasing the probability of beneficial tokens while suppressing harmful outputs. Crucially, it incurs negligible degradation in general task performance.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As the development of large language models (LLMs) rapidly advances, securing these models effectively without compromising their utility has become a pivotal area of research. However, current defense strategies against jailbreak attacks (i.e., efforts to bypass security protocols) often suffer from limited adaptability, restricted general capability, and high cost. To address these challenges, we introduce SafeAligner, a methodology implemented at the decoding stage to fortify defenses against jailbreak attacks. We begin by developing two specialized models: the Sentinel Model, which is trained to foster safety, and the Intruder Model, designed to generate riskier responses. SafeAligner leverages the disparity in security levels between the responses from these models to differentiate between harmful and beneficial tokens, effectively guiding the safety alignment by altering the output token distribution of the target model. Extensive experiments show that SafeAligner can increase the likelihood of beneficial tokens, while reducing the occurrence of harmful ones, thereby ensuring secure alignment with minimal loss to generality.

Problem

• Large Language Models
• Security
• Cost-effectiveness

Innovation

• SafeAligner
• Sentinel model
• Adversary model