On the Suitability of LLM-Driven Agents for Dark Pattern Audits

2026-03-04

AI Summary

This study evaluates the feasibility and limitations of large language model (LLM)-driven agents in detecting dark patterns—such as friction, deception, and coercion—in high-stakes web interfaces, with a focus on data rights request flows under the California Consumer Privacy Act (CCPA). We present the first end-to-end automated auditing framework that integrates LLM-powered autonomous navigation, structured evidence extraction, and a dark pattern classification algorithm. The system is rigorously validated across 456 data broker websites, demonstrating its ability to reliably locate and complete CCPA request processes while accurately categorizing dark patterns. Our findings also delineate the agent’s failure boundaries in complex user interfaces, establishing a reproducible paradigm for large-scale privacy compliance auditing.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Data Mining & Knowledge Management: Knowledge Acquisition from the Web
• Humans and AI: User Experience and Usability

Application Category

• Responsible Web: Consent frameworks and practices on the web
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

As LLM-driven agents begin to autonomously navigate the web, their ability to interpret and respond to manipulative interface design becomes critical. A fundamental question that emerges is: can such agents reliably recognize patterns of friction, misdirection, and coercion in interface design (i.e., dark patterns)? We study this question in a setting where the workflows are consequential: website portals associated with the submission of CCPA-related data rights requests. These portals operationalize statutory rights, but they are implemented as interactive interfaces whose design can be structured to facilitate, burden, or subtly discourage the exercise of those rights. We design and deploy an LLM-driven auditing agent capable of end-to-end traversal of rights-request workflows, structured evidence gathering, and classification of potential dark patterns. Across a set of 456 data broker websites, we evaluate: (1) the ability of the agent to consistently locate and complete request flows, (2) the reliability and reproducibility of its dark pattern classifications, and (3) the conditions under which it fails or produces poor judgments. Our findings characterize both the feasibility and the limitations of using LLM-driven agents for scalable dark pattern auditing.

Problem

• dark patterns
• LLM-driven agents
• interface manipulation
• privacy rights
• web auditing

Innovation

• LLM-driven agents
• dark pattern auditing
• automated web navigation
• CCPA compliance
• interface manipulation detection