Vision Token Manipulation Attacks on Cloud-Edge Inference of Large Vision-Language Models

2026-07-02

AI Summary

This work addresses a critical security vulnerability in cloud-edge collaborative inference for large vision-language models (LVLMs), where visual tokens are susceptible to man-in-the-middle (MitM) attacks during transmission. The study is the first to expose the fragility of visual tokens in this setting and introduces a black-box MitM attack framework that integrates an optimization-driven token selection algorithm with four fundamental attack strategies. Under a stringent budget limiting manipulation to only 10% of visual tokens, the proposed method achieves up to an 88.31% drop in accuracy across six state-of-the-art LVLMs (ranging from 3B to 72B parameters) and four standard benchmarks. These results starkly reveal the severe security risks inherent in current collaborative inference paradigms.

Abstract

Cloud-edge Large Vision-Language Model (LVLM) inference enables efficient deployment by splitting computation between edge devices and cloud servers. In this process, intermediate vision tokens are transmitted from the edge to the cloud over a communication link, thereby exposing a new attack surface. We study vision token manipulation attack (VTM-Attack) under a black-box man-in-the-middle setting, where an adversary intercepts and manipulates a subset of transmitted vision tokens under a budget constraint. We propose four naïve attack strategies and an optimization-based token selection method. Experiments on 6 state-of-the-art LVLMs (3B-72B) across 4 benchmarks show that manipulating only 10\% of vision tokens can reduce accuracy by up to 88.31\%. These results reveal a critical vulnerability in cloud-edge LVLM inference.

Problem

• Vision Token Manipulation
• Cloud-Edge Inference
• Large Vision-Language Models
• Security Vulnerability
• Man-in-the-Middle Attack

Innovation

• Vision Token Manipulation
• Cloud-Edge Inference
• Large Vision-Language Models
• Adversarial Attack
• Black-box Setting