🤖 AI Summary
Existing research on large language models (LLMs) lacks a unified taxonomy for prompt injection and jailbreaking attacks, and insufficiently evaluates defenses under dynamic, interactive scenarios. Method: We propose the first four-dimensional attack taxonomy—spanning prompt-level, model-level, multimodal, and multilingual pathways—and develop a robust alignment framework tailored to interactive settings, alongside a novel automated jailbreaking detection method. We further conduct systematic defense benchmarking, bias diagnosis of existing evaluation benchmarks, and multi-dimensional security measurement. Contribution/Results: Our analysis reveals critical failure modes of current defenses in dynamic interactions, identifies key research gaps—including ethical implications and data bias—and delivers the first comprehensive technical roadmap for LLM safety alignment.
📝 Abstract
Large Language Models (LLMs) have transformed artificial intelligence by advancing natural language understanding and generation, enabling applications across fields beyond healthcare, software engineering, and conversational systems. Despite these advancements in the past few years, LLMs have shown considerable vulnerabilities, particularly to prompt injection and jailbreaking attacks. This review analyzes the state of research on these vulnerabilities and presents available defense strategies. We roughly categorize attack approaches into prompt-based, model-based, multimodal, and multilingual, covering techniques such as adversarial prompting, backdoor injections, and cross-modality exploits. We also review various defense mechanisms, including prompt filtering, transformation, alignment techniques, multi-agent defenses, and self-regulation, evaluating their strengths and shortcomings. We also discuss key metrics and benchmarks used to assess LLM safety and robustness, noting challenges like the quantification of attack success in interactive contexts and biases in existing datasets. Identifying current research gaps, we suggest future directions for resilient alignment strategies, advanced defenses against evolving attacks, automation of jailbreak detection, and consideration of ethical and societal impacts. This review emphasizes the need for continued research and cooperation within the AI community to enhance LLM security and ensure their safe deployment.