Agentic Adversarial Rewriting Exposes Architectural Vulnerabilities in Black-Box NLP Pipelines

2026-04-25

AI Summary

Existing black-box attack methods struggle to effectively evaluate the robustness of multi-component NLP systems under stringent constraints—specifically, binary feedback only, no gradient access, and a query budget of ten or fewer. This work proposes a dual-agent adversarial rewriting framework: an attack agent generates semantics-preserving rewrites, while a prompt optimization agent iteratively refines the attack strategy based solely on binary feedback. The approach achieves the first effective black-box attacks under such strict conditions, revealing critical links between system architecture and vulnerability, and identifying four distinct attack patterns targeting different pipeline stages. Experiments demonstrate evasion rates of 19.95%–40.34% against four LLM-based misinformation detection systems and up to 97.02% against static retrieval systems. Furthermore, defenses informed by these attack patterns reduce evasion rates by as much as 65.18%.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Multiagent Systems: Adversarial Agents
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Multi-component natural language processing (NLP) pipelines are increasingly deployed for high-stakes decisions, yet no existing adversarial method can test their robustness under realistic conditions: binary-only feedback, no gradient access, and strict query budgets. We formalize this strict black-box threat model and propose a two-agent evasion framework operating in a semantic perturbation space. An Attacker Agent generates meaning-preserving rewrites while a Prompt Optimization Agent refines the attack strategy using only binary decision feedback within a 10-query budget. Evaluated against four evidence-based misinformation detection pipelines, the framework achieves evasion rates of 19.95 to 40.34% on modern large language model (LLM) based systems, compared to at most 3.90% for token-level perturbation baselines that rely on surrogate models because they cannot operate under our threat model. A legacy system relying on static lexical retrieval exhibits near-total vulnerability 97.02%, establishing a lower bound that exposes how architectural choices govern the attack surface. Evasion effectiveness is associated with three architectural properties: evidence retrieval mechanism, retrieval-inference coupling, and baseline classification accuracy. The iterative prompt optimization yields the largest marginal gains against the most robust targets, confirming that adaptive strategy discovery is essential when evasion is non-trivial. Analysis of successful rewrites reveals four exploitation patterns, each targeting failures at distinct pipeline stages. A pattern-informed defense reduces the evasion rate by up to 65.18%.

Problem

• black-box NLP pipelines
• adversarial robustness
• architectural vulnerabilities
• semantic perturbation
• evasion attacks

Innovation

• agentic adversarial rewriting
• black-box NLP pipelines
• semantic perturbation
• two-agent evasion framework
• architectural vulnerability