"Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems

2026-02-24

AI Summary

This study investigates human users’ perceptual vulnerability to Agent-Mediated Deception (AMD) attacks in large language model (LLM)-driven agent systems. Leveraging a high-fidelity experimental platform, HAT-Lab, the authors conducted a large-scale empirical study with 303 participants across nine scenarios spanning everyday and professional contexts. The research reveals, for the first time, critical cognitive failure modes among users confronting compromised agents: only 8.6% detected the attack, and experts were paradoxically more susceptible in certain scenarios. To address this vulnerability, the work proposes a novel defense mechanism centered on low-cost verification, workflow-interruptive warnings, and experiential learning. Experimental results demonstrate that over 90% of users significantly enhanced their vigilance after experiential training, confirming the effectiveness and feasibility of the proposed defensive approach.

Technology Category

• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Large language model (LLM) agents are rapidly becoming trusted copilots in high-stakes domains like software development and healthcare. However, this deepening trust introduces a novel attack surface: Agent-Mediated Deception (AMD), where compromised agents are weaponized against their human users. While extensive research focuses on agent-centric threats, human susceptibility to deception by a compromised agent remains unexplored. We present the first large-scale empirical study with 303 participants to measure human susceptibility to AMD. This is based on HAT-Lab (Human-Agent Trust Laboratory), a high-fidelity research platform we develop, featuring nine carefully crafted scenarios spanning everyday and professional domains (e.g., healthcare, software development, human resources). Our 10 key findings reveal significant vulnerabilities and provide future defense perspectives. Specifically, only 8.6% of participants perceive AMD attacks, while domain experts show increased susceptibility in certain scenarios. We identify six cognitive failure modes in users and find that their risk awareness often fails to translate to protective behavior. The defense analysis reveals that effective warnings should interrupt workflows with low verification costs. With experiential learning based on HAT-Lab, over 90% of users who perceive risks report increased caution against AMD. This work provides empirical evidence and a platform for human-centric agent security research.

Problem

• Agent-Mediated Deception
• Human Perception Vulnerability
• LLM Agents
• Trust
• Deception

Innovation

• Agent-Mediated Deception
• Human-Agent Trust
• LLM Agents
• Cognitive Failure Modes
• Experiential Learning