Steal the Patch Size: Adversarially Manipulate Vision-Language Models

2026-06-30

AI Summary

This work proposes a black-box attack method capable of extracting private visual tokenizer configurations—such as image patch size and preprocessing pipelines—from closed-source vision-language models (e.g., Qwen-VL, GPT, Claude) without access to internal model details. The approach exploits a task-level side channel inherent in Vision Transformer (ViT)-style patchification: by constructing grid-aligned images that disrupt natural patch boundaries, the method observes periodic performance degradation and infers tokenizer parameters from these patterns. This study is the first to reveal a practically exploitable side channel arising from patch-based tokenization in vision-language models. Combining grid scanning, padding strategies, and consistency validation, the technique successfully recovers critical tokenizer settings across multiple proprietary models, enabling preprocessing-aware transfer attacks and adversarial manipulation.

Abstract

We present a black-box model-stealing attack that recovers private vision-tokenizer configurations of deployed vision-language models (VLMs), including the visual patch size and input preprocessing pipeline. The key idea is a task-level side channel induced by ViT-style patchification: when a synthetic grid image is aligned with the hidden patch grid, boundary cues are erased at tokenization, causing periodic accuracy drop. By sweeping the grid cell size and measuring these collapses, we infer the patch size; by introducing padding and a consistency-check test, we further identify whether preprocessing is dynamic- or fixed-resolution and recover the target resize resolution. Across open-source Qwen-VL variants and proprietary models including GPT and Claude, we reliably recover tokenizer-related parameters. Finally, we show that such leakage enables preprocessing-aware transfer attacks and model-targeted adversarial manipulation.

Problem

• model stealing
• vision-language models
• patch size
• tokenizer configuration
• black-box attack

Innovation

• model stealing
• vision-language models
• patch size inference
• side-channel attack
• adversarial manipulation