red teaming

This study addresses the high false-positive rate and low manual verification efficiency in vulnerability assessment (VA) reports. We propose and implement, for the first time, an LLM-driven paradigm for automatic VA report validation. Methodologically, we integrate critical literature review, semantic parsing of VA reports, and credibility reasoning to construct an LLM-based verification framework tailored to offensive security contexts. Unlike conventional rule-based or statistical models, our approach enables deep discrimination of technical vulnerability details, contextual consistency, and evidentiary sufficiency. Experimental results demonstrate a 37.2% average reduction in false positives, a 5.8× improvement in verification throughput, and an accuracy of 92.4%, substantially reducing human effort. This work bridges a critical research gap—applying LLMs to trustworthy, evidence-grounded validation in offensive cybersecurity tasks.

AI safety evaluation lacks consensus standards, limiting its utility for governance and policy decisions. This paper introduces the first practical AI safety evaluation framework, systematically integrating threat modeling, assessment design, and validity validation. It formally defines three essential criteria for “useful” evaluations—risk alignment, reproducibility, and scalability—along with associated quantitative parameters. Innovatively distinguishing formal metrics from real-world risk coverage, the framework establishes an evolutionary paradigm—from isolated tests to modular, composable evaluation suites. It synergistically integrates red-teaming, evaluation validity analysis, and cybersecurity best practices to jointly optimize reliability, construct validity, and operational feasibility. The resulting safety evaluation guidelines have been adopted by industry stakeholders and policymaking bodies, demonstrably enhancing the interpretability of evaluation outcomes and their actionable support for risk-informed decision-making.

This work exposes a critical security vulnerability in text-to-image models: when backdoored, they can be maliciously triggered to generate unsafe images—even from benign prompts such as “a photo of a cat.” Unlike prior studies focusing on passive exploitation, we propose the first *active* paradigm for unsafe image generation targeting benign prompts. We design a stealthy, concept-decoupled poisoning method that imposes constrained optimization directly in the text embedding space, enabling high-precision triggering on target prompts while maintaining strong robustness against non-target prompts. Evaluated on mainstream diffusion models—including Stable Diffusion—our approach achieves a trigger success rate exceeding 92% and a false-trigger rate below 5% on non-target prompts, significantly reducing unintended side effects. This work establishes a novel framework for security assessment and defense of generative AI models, providing both conceptual insight and practical technical foundations.

Attack scenario descriptions in cybersecurity automation lack formal semantic foundations, hindering systematic analysis and automation. Method: This paper proposes an abstract, formal model based on UML class diagrams, enabling the first unified modeling of attack context and attack scenarios. The model supports structured input, automated processing, and cross-process reuse, directly facilitating two core tasks: attack analysis and automated attack script generation. Contribution/Results: Evaluated on real-world attack analysis and cybersecurity training script generation, the model demonstrates strong feasibility and effectiveness. It fills a critical gap in formal attack scenario modeling and establishes a scalable, verifiable semantic foundation for security process automation—enhancing interoperability, reproducibility, and formal reasoning in cyber defense systems.

Although state-of-the-art large language models employ output-level safety mechanisms, they may still inadvertently leak harmful knowledge through indirect prompting, enabling open-source models—after fine-tuning—to reconstruct hazardous capabilities, thereby posing ecosystem-level risks. This work presents the first systematic investigation of such cross-model capability transfer threats and introduces a three-stage elicitation attack framework: by crafting benign prompts that are semantically proximate to harmful tasks, adversaries can extract implicit hazardous information from safeguarded models and use it to fine-tune open-source counterparts. Experiments on dangerous chemical synthesis tasks demonstrate that this approach recovers approximately 40% of the performance gap between protected and unrestricted models, with attack efficacy significantly amplified by both the capability of the frontier model and the scale of fine-tuning data, thereby challenging the adequacy of current safety paradigms.

This work addresses the limitations of existing adversarial simulation tools, which rely on agent-based instrumentation of target systems, often leaving anomalous artifacts and failing to faithfully replicate human attacker behavior—particularly in critical phases of the cyber kill chain such as initial access and interactive operations. To overcome these shortcomings, the authors propose and implement an open-source attack scripting language coupled with an agentless execution engine that closely emulates real-world attacker tactics. This approach enables high-fidelity, interactive simulation of complete kill chain stages, including initial access, privilege escalation, and lateral movement. Experimental results demonstrate that system logs generated by this method exhibit significantly greater behavioral similarity to those produced by actual human-driven attacks, thereby enhancing the realism and effectiveness of security testing and intrusion detection research.

This work addresses the limitations of existing defense methods that analyze either user prompts or model outputs in isolation, rendering them ineffective against composite attacks where malicious intent is concealed in the prompt and harmful effects manifest only in the response. To overcome this, the paper introduces the first joint verification framework that concurrently evaluates prompt intent and output harm before response generation, establishing a unified threat model for prompt-response pairs. The framework employs a multi-agent collaborative architecture comprising specialized intent and harm analyzers, a conflict-resolution Judge module, and a chain-of-thought reasoning mechanism, with tailored verification strategies designed for five threat categories: jailbreaking, prompt injection, phishing, network abuse, and harmful content. Experiments demonstrate that the proposed method achieves an average F1 score of 0.95 across multiple benchmarks, reduces attack success rates to 4.1%, significantly outperforms the strongest baseline, and maintains robustness against architecture-aware adaptive attacks.

This work proposes a general-purpose red-teaming framework that overcomes the limitations of existing automated approaches, which are often confined to specific security scenarios and rely on evaluators known during training, thereby lacking generalization to novel adversarial targets. By end-to-end fine-tuning compact language models such as Qwen3-8B and integrating multi-objective adversarial example generation with adaptive optimization strategies, the method generates effective attacks against arbitrary red-teaming tasks without requiring predefined evaluators. Experimental results demonstrate significant improvements in attack generation performance both within and across domains. To the best of our knowledge, this is the first approach to achieve evaluator-agnostic, generalizable red-teaming automation, effectively transcending the constraints of conventional methods in terms of task scope and adaptability.

This study addresses two critical adversarial threats to large language models (LLMs): prompt injection and goal hijacking. We systematically evaluate adversarial fine-tuning as a defense mechanism—first demonstrating its efficacy against such attacks—by constructing a comprehensive benchmark of diverse adversarial examples and empirically assessing GPT-3 variants alongside contemporaneous state-of-the-art LLMs. Results show that, without mitigation, attack success rates reach 31%; after adversarial fine-tuning, the success rate drops to near zero for smaller GPT-3 models, confirming the method’s strong defensive capability. Crucially, we uncover a robust size–security trade-off: larger models (e.g., Davinci) exhibit significantly higher vulnerability. Our work establishes a reproducible defense paradigm for enhancing LLM robustness and provides foundational empirical evidence for adversarial resilience in modern language models.

Vision-language models (VLMs) exhibit critical security vulnerabilities to text prompt injection attacks in open-ended interactive settings. Method: We propose a lightweight, efficient, and generalizable attack method that synergistically combines prompt engineering with adversarial input construction—requiring no model gradients or fine-tuning, and relying solely on carefully crafted textual instructions jointly perturbed with image context to mislead mainstream VLMs (e.g., LLaVA, Qwen-VL). Contribution/Results: Our approach achieves significantly higher attack success rates across multi-task benchmarks—averaging a 12.7% improvement over state-of-the-art baselines—while maintaining sub-0.8-second per-attack latency and reducing GPU memory consumption by over 60%. This work constitutes the first systematic empirical demonstration of VLMs’ susceptibility to pure-text prompt injection under realistic open-world interaction protocols, thereby establishing a foundational benchmark and technical reference for future robustness modeling and defensive mechanism design.

This study addresses the vulnerability of large language models to prompt injection attacks when sensitive information is embedded in system prompts, which can lead to unintended secret disclosure. The authors propose an adaptive adversarial attack framework that dynamically evolves attack strategies over more than 20,000 red-team evaluations to systematically assess nine state-of-the-art defense mechanisms. Experimental results demonstrate that all defenses relying solely on the model’s intrinsic safeguards fail to prevent information leakage, whereas only application-layer output filtering combined with hard-coded rules achieves zero leakage. This work provides the first large-scale empirical evidence—through adaptive adversarial testing—that security boundaries must be enforced by application code rather than by the model itself.