Shadows in the Code: Exploring the Risks and Defenses of LLM-based Multi-Agent Software Development Systems

2025-11-23

AI Summary

This paper identifies two novel, stealthy security risks in large language model (LLM)-based multi-agent software development systems: malicious users interacting with benign agents (MU-BA) and benign users interacting with malicious agents (BU-MA). Method: We formally define these threat models, propose the Implicit Malicious Behavior Injection Attack (IMBIA) framework, and design Adv-IMBIA—a defense mechanism integrating adversarial prompt injection, fine-grained behavioral monitoring, and critical-node protection. Evaluation is conducted across ChatDev, MetaGPT, and AgentVerse. Results: IMBIA achieves up to 93% attack success rate across all three frameworks; Adv-IMBIA significantly mitigates attacks, especially in MU-BA scenarios. Our analysis systematically pinpoints critical vulnerabilities in coding and testing phases, establishing the first theoretical security framework and practical defense paradigm for LLM-powered multi-agent systems.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The rapid advancement of Large Language Model (LLM)-driven multi-agent systems has significantly streamlined software developing tasks, enabling users with little technical expertise to develop executable applications. While these systems democratize software creation through natural language requirements, they introduce significant security risks that remain largely unexplored. We identify two risky scenarios: Malicious User with Benign Agents (MU-BA) and Benign User with Malicious Agents (BU-MA). We introduce the Implicit Malicious Behavior Injection Attack (IMBIA), demonstrating how multi-agent systems can be manipulated to generate software with concealed malicious capabilities beneath seemingly benign applications, and propose Adv-IMBIA as a defense mechanism. Evaluations across ChatDev, MetaGPT, and AgentVerse frameworks reveal varying vulnerability patterns, with IMBIA achieving attack success rates of 93%, 45%, and 71% in MU-BA scenarios, and 71%, 84%, and 45% in BU-MA scenarios. Our defense mechanism reduced attack success rates significantly, particularly in the MU-BA scenario. Further analysis reveals that compromised agents in the coding and testing phases pose significantly greater security risks, while also identifying critical agents that require protection against malicious user exploitation. Our findings highlight the urgent need for robust security measures in multi-agent software development systems and provide practical guidelines for implementing targeted, resource-efficient defensive strategies.

Problem

• Security risks in LLM-based multi-agent software development systems
• Malicious behavior injection attacks on multi-agent development frameworks
• Vulnerability patterns across different agent collaboration scenarios

Innovation

• Introducing Implicit Malicious Behavior Injection Attack
• Proposing Adv-IMBIA as defense mechanism
• Evaluating vulnerabilities across multiple agent frameworks