Don't Click That: Teaching Web Agents to Resist Deceptive Interfaces

2026-05-10

AI Summary

This work addresses the vulnerability of vision-language model–driven web agents to deceptive user interfaces, which often compromises their task execution due to insufficient defensive capabilities. To bridge this gap, the authors propose DUDE, a two-stage framework that formalizes deception-aware defense mechanisms for the first time. DUDE integrates hybrid-reward reinforcement learning, an asymmetric penalty strategy, and experience-summary distillation to distill failure patterns into transferable defensive guidance. Evaluated on RUC—a newly curated benchmark comprising 1,407 deceptive scenarios—DUDE reduces agent sensitivity to deceptive interfaces by 53.8% while preserving task performance, thereby achieving a synergistic optimization of robustness and efficacy.

Abstract

Vision-language model (VLM) based web agents demonstrate impressive autonomous GUI interaction but remain vulnerable to deceptive interface elements. Existing approaches either detect deception without task integration or document attacks without proposing defenses. We formalize deception-aware web agent defense and propose DUDE (Deceptive UI Detector & Evaluator), a two-stage framework combining hybrid-reward learning with asymmetric penalties and experience summarization to distill failure patterns into transferable guidance. We introduce RUC (Real UI Clickboxes), a benchmark of 1,407 scenarios spanning four domains and deception categories. Experiments show DUDE reduces deception susceptibility by 53.8% while maintaining task performance, establishing an effective foundation for robust web agent deployment.

Problem

• deceptive interfaces
• web agents
• vision-language models
• GUI interaction
• robustness

Innovation

• deception-aware agents
• hybrid-reward learning
• asymmetric penalties
• experience summarization
• web agent robustness