MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignment

2025-05-29

AI Summary

This work addresses the emerging “False Benign Attack” (FBA) in Model Context Protocol (MCP)—a security threat wherein malicious online content, without requiring file downloads, deceives AI agents into executing unauthorized actions on the user side. We propose RAG-Pref, a novel paradigm that tightly integrates Retrieval-Augmented Generation (RAG) with preference alignment. We are the first to identify and empirically demonstrate the critical role of post-training methods—particularly GRPO—in enhancing model refusal capability against FBA. Additionally, we release the first FBA-specific preference dataset. Experiments show that combining RAG-Pref with Direct Preference Optimization (DPO) significantly improves large language models’ detection and refusal rates for FBA, achieving strong robustness and scalability. Our approach establishes a new, practical pathway for security-aligned inference in MCP ecosystems.

Technology Category

• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Adversarial Agents
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Large-scale security measurements

Abstract

The model context protocol (MCP) has been widely adapted as an open standard enabling the seamless integration of generative AI agents. However, recent work has shown the MCP is susceptible to retrieval-based"falsely benign"attacks (FBAs), allowing malicious system access and credential theft, but requiring that users download compromised files directly to their systems. Herein, we show that the threat model of MCP-based attacks is significantly broader than previously thought, i.e., attackers need only post malicious content online to deceive MCP agents into carrying out their attacks on unsuspecting victims' systems. To improve alignment guardrails against such attacks, we introduce a new MCP dataset of FBAs and (truly) benign samples to explore the effectiveness of direct preference optimization (DPO) for the refusal training of large language models (LLMs). While DPO improves model guardrails against such attacks, we show that the efficacy of refusal learning varies drastically depending on the model's original post-training alignment scheme--e.g., GRPO-based LLMs learn to refuse extremely poorly. Thus, to further improve FBA refusals, we introduce Retrieval Augmented Generation for Preference alignment (RAG-Pref), a novel preference alignment strategy based on RAG. We show that RAG-Pref significantly improves the ability of LLMs to refuse FBAs, particularly when combined with DPO alignment, thus drastically improving guardrails against MCP-based attacks.

Problem

• MCP is vulnerable to online-posted falsely benign attacks
• Existing alignment methods vary in effectiveness for refusal training
• RAG-Pref enhances LLMs' ability to refuse falsely benign exploits

Innovation

• Uses Direct Preference Optimization (DPO) for refusal training
• Introduces Retrieval Augmented Generation for Preference alignment (RAG-Pref)
• Combines RAG-Pref with DPO to enhance attack refusal