Rethinking MCP Security: A Large-Scale Study of Runtime MCP Servers and Security Scanner Reliability

2026-07-13

AI Summary

This study addresses the lack of large-scale empirical security evaluation of MCP servers in real-world environments, which has traditionally relied on unreliable scanning tools. To bridge this gap, the authors introduce MCPZoo—the first dataset enabling large-scale dynamic analysis of MCP servers, encompassing 64,611 unique services. MCPZoo leverages a multi-agent automation framework to transform static code into interactive, executable instances, integrating environment inference, feedback-driven repair, and protocol-level runtime validation. The research reveals, for the first time at an ecosystem scale, that mainstream scanners exhibit true positive rates below 50% and produce highly inconsistent results. To facilitate practical risk assessment, the authors provide a public query interface to the dataset.

Abstract

The Model Context Protocol (MCP) has rapidly established itself as a standard interface for enabling LLM-based agents to interact with external tools and services. As MCP servers are increasingly entrusted with security-sensitive operations, understanding their real-world risks has become critical. In practice, due to the absence of large-scale runtime MCP servers, such understanding largely relies on security scanners applied to a small number of cases, yet the reliability of these assessments remains unclear. In this study, we revisit how MCP security is measured. We present MCPZoo, the largest collection of MCP servers for dynamic analysis to date. MCPZoo is constructed through a multi-agent framework for transforming in-the-wild static repositories into dynamic services. The framework emulates how human experts build, diagnose, and iteratively repair deployment and runtime defects by combining environment inference with feedback-driven refinement. To ensure practical interactivity at runtime, the servers are validated via real protocol interactions. As a result, MCPZoo contains 64,611 unique MCP servers (113,927 in total), with more than 37,288 supporting dynamic analysis. Leveraging MCPZoo, we conduct the first ecosystem-scale measurement of MCP servers and the scanners that analyze them. While existing scanners report that 96.89% of servers are risky, we find that these signals are unreliable. In particular, manual validation shows that less than 50% of sampled alerts are true positives, and scanner outputs exhibit clear inconsistency across scanners. Overall, MCPZoo enables large-scale, reproducible measurement of MCP server security and exposes limitations of current scanning practices. We further release a public query interface to support practical risk assessment of MCP servers.

Problem

• Model Context Protocol
• MCP security
• security scanners
• runtime analysis
• large-scale study

Innovation

• MCPZoo
• dynamic analysis
• multi-agent framework
• security scanner reliability
• Model Context Protocol