A Penny for Your Prompts: Experiments Detecting and Mitigating LLM Usage by Survey Respondents

2026-07-01

AI Summary

This study addresses the growing threat to survey data validity posed by respondents using large language models (LLMs) to complete questionnaires on crowdsourcing platforms. Through multiple controlled experiments, it systematically quantifies a stark disparity in LLM usage rates between two major platforms—below 10% on Prolific versus over 80% on Mechanical Turk—and introduces an active screening approach combining keystroke logging with AI-directed questions. The research evaluates interventions such as disabling copy-paste functionality, instructional warnings, and detection of AI-generated response patterns, finding that while these measures reduce LLM usage, their impact on overall data quality remains limited. This work provides an empirical foundation and a practical detection framework for safeguarding the authenticity of crowdsourced survey data.

Abstract

Large language models are increasingly used by participants on crowdsourcing platforms when responding to surveys, potentially undermining the validity of collected data. Our study aims to quantify the prevalence of this behavior and investigate methods to detect and prevent it. In a series of surveys (N = 250), we examined conditions such as platform choice, survey length, requests not to use AI, and disabling copy-paste functionality. We were able to identify distinct characteristics of LLM-assisted responses and found that their frequency varied widely, from under 10% on Prolific to over 80% on Mechanical Turk. Mitigation measures reduced LLM usage but did not necessarily improve data quality. No participants employed browser-use agents at the time of our survey, but we report on our own detection experiments. We recommend that researchers actively screen survey responses for LLM usage by recording and analyzing keystroke data and crafting instructions and questions aimed at AI.

Problem

• large language models
• survey validity
• crowdsourcing
• LLM usage
• data quality

Innovation

• LLM detection
• survey validity
• keystroke analysis
• crowdsourcing platforms
• AI mitigation