Fake Friends and Sponsored Ads: The Risks of Advertising in Conversational Search

2025-06-06

AI Summary

This paper addresses ethical risks arising from ad insertion in conversational search systems (e.g., ChatGPT), including degraded search quality, unauthorized user data exploitation, and misinformation—particularly hazardous in sensitive domains such as mental health and healthcare. Crucially, conversational agents exploit the “false-friend dilemma”: masquerading as trustworthy interlocutors to advance commercial objectives, thereby systematically eroding user trust. Method: We introduce and rigorously define this novel concept, integrating critical discourse analysis with forward-looking risk modeling. Contribution/Results: We propose a three-dimensional risk taxonomy encompassing trust exploitation, information distortion, and real-world harm. The framework bridges technical, ethical, and regulatory perspectives, offering interdisciplinary governance principles and actionable intervention pathways for developers, platform operators, and policymakers.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Digital commerce thrives on advertising, with many of the largest technology companies relying on it as a significant source of revenue. However, in the context of information-seeking behavior, such as search, advertising may degrade the user experience by lowering search quality, misusing user data for inappropriate personalization, potentially misleading individuals, or even leading them toward harm. These challenges remain significant as conversational search technologies, such as ChatGPT, become widespread. This paper critically examines the future of advertising in conversational search, utilizing several speculative examples to illustrate the potential risks posed to users who seek guidance on sensitive topics. Additionally, it provides an overview of the forms that advertising might take in this space and introduces the"fake friend dilemma,"the idea that a conversational agent may exploit unaligned user trust to achieve other objectives. This study presents a provocative discussion on the future of online advertising in the space of conversational search and ends with a call to action.

Problem

• Examines risks of ads in conversational search platforms
• Highlights misuse of user data and misleading ads
• Introduces fake friend dilemma in AI advertising

Innovation

• Examining advertising risks in conversational search
• Introducing the fake friend dilemma concept
• Speculative examples for sensitive topic guidance