How Generative AI Empowers Attackers and Defenders Across the Trust & Safety Landscape

2025-11-10

Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems

AI Summary

This study addresses the dual role of generative artificial intelligence in Trust & Safety, which simultaneously lowers the barrier to producing harmful content—such as child sexual exploitation material, election manipulation, hate speech, scams, and extremist propaganda—and enhances defensive capabilities through scalable detection, counter-narrative dissemination, investigative support, and improved content moderator well-being. Despite its significant implications, the mechanisms underlying this duality remain underexplored. Drawing on in-depth interviews with 43 domain experts, this work proposes a strategic analytical framework that elucidates how generative AI reshapes adversarial and protective dynamics across four key operational contexts: content generation, identification, investigation, and user support. The findings clarify both the risks and opportunities inherent in deploying generative AI within Trust & Safety ecosystems.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Safety and Robustness
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Generative AI (GenAI) is a powerful technology poised to reshape Trust & Safety. While misuse by attackers is a growing concern, its defensive capacity remains underexplored. This paper examines these effects through a qualitative study with 43 Trust & Safety experts across five domains: child safety, election integrity, hate and harassment, scams, and violent extremism. Our findings characterize a landscape in which GenAI empowers both attackers and defenders. GenAI dramatically increases the scale and speed of attacks, lowering the barrier to entry for creating harmful content, including sophisticated propaganda and deepfakes. Conversely, defenders envision leveraging GenAI to detect and mitigate harmful content at scale, conduct investigations, deploy persuasive counternarratives, improve moderator wellbeing, and offer user support. This work provides a strategic framework for understanding GenAI’s impact on Trust & Safety and charts a path for its responsible use in creating safer online environments.

Problem

• Generative AI
• Trust & Safety
• online harms
• deepfakes
• misuse

Innovation

• Generative AI
• Trust & Safety
• dual-use technology
• harmful content detection
• deepfakes