Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses

2025-04-02

AI Summary

This work investigates the evolutionary patterns and key determinants of large language model (LLM) security, specifically focusing on robustness against jailbreak attacks. Method: We conduct a systematic, cross-model evaluation across 12 prominent open- and closed-source LLMs—spanning diverse architectures, parameter scales, and versions—and integrate four state-of-the-art jailbreak attack methods (including GCG and AutoDAN) with three novel defense strategies: prompt sanitization, response recalibration, and multi-model arbitration. Contribution/Results: Our study is the first to empirically demonstrate that newer model versions are not inherently more secure; that optimized smaller models can outperform unprotected larger ones; and that synergistic defense combinations reduce average attack success rates by 62%. We propose a plug-and-play, lightweight defense framework and validate the generalizability of combinatorial defense gains—providing empirical foundations and actionable engineering pathways for advancing LLM security.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large Language Models (LLMs) are increasingly popular, powering a wide range of applications. Their widespread use has sparked concerns, especially through jailbreak attacks that bypass safety measures to produce harmful content. In this paper, we present a comprehensive security analysis of large language models (LLMs), addressing critical research questions on the evolution and determinants of model safety. Specifically, we begin by identifying the most effective techniques for detecting jailbreak attacks. Next, we investigate whether newer versions of LLMs offer improved security compared to their predecessors. We also assess the impact of model size on overall security and explore the potential benefits of integrating multiple defense strategies to enhance model robustness. Our study evaluates both open-source models (e.g., LLaMA and Mistral) and closed-source systems (e.g., GPT-4) by employing four state-of-the-art attack techniques and assessing the efficacy of three new defensive approaches.

Problem

• Analyzing jailbreak attack detection techniques in LLMs
• Evaluating security improvements in newer LLM versions
• Assessing model size impact on LLM defense strategies

Innovation

• Detecting jailbreak attacks effectively
• Evaluating LLM version security improvements
• Integrating multiple defense strategies