SnapGuard: Lightweight Prompt Injection Detection for Screenshot-Based Web Agents

2026-04-28

AI Summary

This work addresses the high computational cost and inefficiency of existing prompt injection detection methods that rely on large vision-language models. The authors propose a lightweight detection mechanism that, for the first time, operates without depending on such large models. By analyzing multimodal features extracted from webpage screenshots—including anomalous visual gradient distributions, action-oriented textual signals derived from contrast polarity inversion recovery, and multimodal representation learning—the method efficiently identifies malicious content. Evaluated across eight attack types and two benign scenarios, the approach achieves an F1 score of 0.75, demonstrates an 8× faster inference speed than GPT-4o-prompt (1.81 seconds versus 14.50 seconds), and incurs no additional memory overhead.

Abstract

Web agents have emerged as an effective paradigm for automating interactions with complex web environments, yet remain vulnerable to prompt injection attacks that embed malicious instructions into webpage content to induce unintended actions. This threat is further amplified for screenshot-based web agents, which operate on rendered visual webpages rather than structured textual representations, making predominant text-centric defenses ineffective. Although multimodal detection methods have been explored, they often rely on large vision-language models (VLMs), incurring significant computational overhead. The bottleneck lies in the complexity of modern webpages: VLMs must comprehend the global semantics of an entire page, resulting in substantial inference time and GPU memory usage. This raises a critical question: can we detect prompt injection attacks from screenshots in a lightweight manner? In this paper, we observe that injected webpages exhibit distinct characteristics compared to benign ones from both visual and textual perspectives. Building on this insight, we propose SnapGuard, a lightweight yet accurate method that reformulates prompt injection detection as multimodal representation analysis over webpage screenshots. SnapGuard leverages two complementary signals: a visual stability indicator that identifies abnormally smooth gradient distributions induced by malicious content, and action-oriented textual signals recovered via contrast-polarity reversal. Extensive evaluations across eight attacks and two benign settings demonstrate that SnapGuard achieves an F1 score of 0.75, outperforming GPT-4o-prompt while being 8x faster (1.81s vs. 14.50s) and introducing no additional memory overhead.

Problem

• prompt injection
• web agents
• screenshot-based
• lightweight detection
• multimodal

Innovation

• prompt injection detection
• screenshot-based web agents
• lightweight multimodal analysis
• visual stability indicator
• contrast-polarity reversal