Verifying Intent and Harm: A Unified Defense Against LLM-Generated Threats

2026-06-24

AI Summary

This work addresses the limitations of existing defense methods that analyze either user prompts or model outputs in isolation, rendering them ineffective against composite attacks where malicious intent is concealed in the prompt and harmful effects manifest only in the response. To overcome this, the paper introduces the first joint verification framework that concurrently evaluates prompt intent and output harm before response generation, establishing a unified threat model for prompt-response pairs. The framework employs a multi-agent collaborative architecture comprising specialized intent and harm analyzers, a conflict-resolution Judge module, and a chain-of-thought reasoning mechanism, with tailored verification strategies designed for five threat categories: jailbreaking, prompt injection, phishing, network abuse, and harmful content. Experiments demonstrate that the proposed method achieves an average F1 score of 0.95 across multiple benchmarks, reduces attack success rates to 4.1%, significantly outperforms the strongest baseline, and maintains robustness against architecture-aware adaptive attacks.

Abstract

Large language models (LLMs) are increasingly deployed in interactive applications, yet they remain vulnerable to adversarial interactions that induce harmful, deceptive, or policy-violating outputs. Existing defenses typically analyze either user prompts or generated outputs, but not both. However, many real-world attacks exploit a separation between adversarial intent expressed in the prompt and actionable harm manifested only in the response. As a result, prompt-only and response-only defenses frequently miss unsafe interactions that appear benign when viewed from either side in isolation. We present a verification-centric defense framework that jointly evaluates prompt intent and response harm before an LLM response is delivered to a user. The framework employs specialized analysts for intent and harm assessment together with a Judge for conflict resolution. We formalize a threat model for prompt-response attacks and evaluate the framework across five threat categories: jailbreaks, prompt injection, phishing, cyber abuse, and harmful content. Experiments on multiple benchmark datasets show that jointly verifying prompt intent and response harm consistently outperforms single-sided defenses and single-agent reasoning baselines. Across threat categories, the framework improves average F1 from 0.90 for the strongest applicable baselines to 0.95 while reducing the average attack success rate to 4.1 percent. Compared with a Single-Agent+CoT baseline, it improves average F1 from 0.87 to 0.95 and reduces the false positive rate on benign-sensitive requests from 0.12 to 0.06. We further evaluate architecture-aware adaptive attacks in which the attacker knows the verifier structure and attempts to bypass individual verification components. Our results suggest that prompt-response verification provides a practical foundation for securing LLM applications against evolving adversarial threats.

Problem

• large language models
• adversarial attacks
• prompt-response verification
• harmful content
• security defense

Innovation

• prompt-response verification
• intent-harm joint analysis
• LLM security
• adversarial defense
• unified verification framework