Prevalence of Security and Privacy Risk-Inducing Usage of AI-based Conversational Agents

2025-10-31

AI Summary

This study investigates the prevalence and user awareness gaps concerning security and privacy risks arising from end-user behaviors in daily interactions with AI dialogue agents (e.g., ChatGPT, Gemini). Drawing on a nationally representative survey of 3,270 UK adults recruited via Prolific, it provides the first large-scale empirical validation of academic threat models—including uploading sensitive files, disclosing personally identifiable information, and jailbreaking attempts. Results show that 33% of regular users engage in high-risk behaviors potentially exploitable by adversaries; 25% have attempted jailbreaking; and over half remain unaware that their inputs may be used for model training. The study’s key contribution lies in bridging theoretical threat modeling with real-world behavioral evidence at scale, revealing a critical misalignment between user practices and vendor safeguards. These findings offer empirically grounded insights to inform enhanced data transparency, human-centered safety nudges, and robust AI system defense strategies.

Technology Category

• Philosophy and Ethics of AI: Privacy & Security
• Humans and AI: User Experience and Usability
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Data transparency and provenance
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent improvement gains in large language models (LLMs) have lead to everyday usage of AI-based Conversational Agents (CAs). At the same time, LLMs are vulnerable to an array of threats, including jailbreaks and, for example, causing remote code execution when fed specific inputs. As a result, users may unintentionally introduce risks, for example, by uploading malicious files or disclosing sensitive information. However, the extent to which such user behaviors occur and thus potentially facilitate exploits remains largely unclear. To shed light on this issue, we surveyed a representative sample of 3,270 UK adults in 2024 using Prolific. A third of these use CA services such as ChatGPT or Gemini at least once a week. Of these ``regular users'', up to a third exhibited behaviors that may enable attacks, and a fourth have tried jailbreaking (often out of understandable reasons such as curiosity, fun or information seeking). Half state that they sanitize data and most participants report not sharing sensitive data. However, few share very sensitive data such as passwords. The majority are unaware that their data can be used to train models and that they can opt-out. Our findings suggest that current academic threat models manifest in the wild, and mitigations or guidelines for the secure usage of CAs should be developed. In areas critical to security and privacy, CAs must be equipped with effective AI guardrails to prevent, for example, revealing sensitive information to curious employees. Vendors need to increase efforts to prevent the entry of sensitive data, and to create transparency with regard to data usage policies and settings.

Problem

• Assessing prevalence of risky user behaviors with AI conversational agents
• Investigating security vulnerabilities like jailbreaks and data exposure
• Evaluating user awareness of data usage policies and safeguards

Innovation

• Surveyed user behaviors with AI conversational agents
• Identified prevalent security and privacy risks
• Proposed enhanced guardrails and transparency measures