MIRROR: Novelty-Constrained Memory-Guided MCTS Red-Teaming for Agentic RAG

2026-06-25

AI Summary

This work addresses the vulnerability of multimodal agent-based RAG systems to cross-surface attacks—including text poisoning, image injection, query tampering, and tool manipulation—highlighting the limitations of existing red-teaming methods that are confined to single modalities and reliant on known attack templates. To overcome these shortcomings, the authors propose a unified cross-surface red-teaming framework that integrates memory-guided Monte Carlo Tree Search (MCTS) with Retrieval-Augmented Generation (RAG), incorporating an explicit novelty constraint during attack generation. A deterministic novelty gating mechanism filters out candidate attacks matching historical patterns, thereby ensuring diversity and generalization. Empirical results demonstrate a 76% success rate in image poisoning attacks (versus 52% for the baseline), a 97% attack success rate (ASR) on orchestrator components with 50% lower query cost, and the lowest coefficient of variation (0.47) in cross-surface attack success rates. The study also introduces ART-SafeBench, a benchmark comprising 41,000 annotated records.

Abstract

Multimodal agentic retrieval-augmented generation (RAG) systems expand the attack surface beyond prompt injection to include text poisoning, image injection, direct-query attacks, and orchestrator-level tool manipulation. Existing red-teaming approaches are typically surface-specific and often recycle known attack templates; on text-poisoning benchmarks we measure 73-84% exact duplication. We present MIRROR, a unified cross-surface framework that performs memory-guided Monte Carlo tree search while conditioning candidate generation on retrieved context under an explicit novelty constraint. A deterministic Novelty Gate rejects any candidate matching the retrieval set under normalized comparison, allowing retrieval to inform search priors without enabling prompt copying. Across four attack surfaces on a multimodal agentic RAG target, MIRROR attains 76% ASR on image poisoning compared with 52% for baselines, 97% ASR on orchestrator attacks at half the query cost, and the lowest cross-surface variance (coefficient of variation 0.47). In contrast, specialized baselines collapse across surfaces: suffix optimization reaches 79% ASR on text poisoning but 1% on direct queries. We release ART-SafeBench with 41,815 in-package records and runtime adapters yielding 41,991+ total records across four surfaces.

Problem

• red-teaming
• retrieval-augmented generation
• multimodal agentic RAG
• attack surface
• novelty constraint

Innovation

• Memory-Guided MCTS
• Novelty Constraint
• Cross-Surface Red-Teaming
• Retrieval-Augmented Generation
• Agentic RAG