TrojanRobot: Physical-World Backdoor Attacks Against VLM-based Robotic Manipulation

2024-11-18

AI Summary

This work introduces the first high-stealth, real-world-deployable backdoor attack against vision-language model (VLM)-driven physical robotic systems. Addressing critical challenges in practical settings—namely low attack stealth and poor generalizability—the authors propose a modular injection-based perception poisoning mechanism and establish the novel “LVLM-as-a-backdoor” paradigm. They design three fine-grained attack variants: token permutation, action stalling, and intent manipulation. The method integrates module-level poisoning, VLM-specific backdoor fine-tuning, and seamless deployment on a physical robotic platform (UR3e). Extensive evaluation across four state-of-the-art VLMs and 18 real-world manipulation tasks demonstrates high attack success rates, strong cross-environment transferability, and imperceptible behavioral perturbations. All experimental videos and source code are publicly released.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Robotic manipulation in the physical world is increasingly empowered by extit{large language models} (LLMs) and extit{vision-language models} (VLMs), leveraging their understanding and perception capabilities. Recently, various attacks against such robotic policies have been proposed, with backdoor attacks drawing considerable attention for their high stealth and strong persistence capabilities. However, existing backdoor efforts are limited to simulators and suffer from physical-world realization. To address this, we propose extit{TrojanRobot}, a highly stealthy and broadly effective robotic backdoor attack in the physical world. Specifically, we introduce a module-poisoning approach by embedding a backdoor module into the modular robotic policy, enabling backdoor control over the policy's visual perception module thereby backdooring the entire robotic policy. Our vanilla implementation leverages a backdoor-finetuned VLM to serve as the backdoor module. To enhance its generalization in physical environments, we propose a prime implementation, leveraging the LVLM-as-a-backdoor paradigm and developing three types of prime attacks, ie, extit{permutation}, extit{stagnation}, and extit{intentional} attacks, thus achieving finer-grained backdoors. Extensive experiments on the UR3e manipulator with 18 task instructions using robotic policies based on four VLMs demonstrate the broad effectiveness and physical-world stealth of TrojanRobot. Our attack's video demonstrations are available via a github link url{https://trojanrobot.github.io}.

Problem

• Adversarial Attacks
• Large Language Models
• Visual Understanding

Innovation

• Trojan Robots
• Stealth Control
• Adversarial Perturbations