Organization Matters: A Qualitative Study of Organizational Dynamics in Red Teaming Practices For Generative AI

2025-08-17

AI Summary

This study addresses the limited practical efficacy of generative AI (GenAI) red teaming by uncovering organizational barriers—including siloed departments, delayed risk awareness, and marginalization of red teams—that systematically impede risk identification. Through semi-structured interviews and qualitative analysis, it is the first to systematically identify non-technical, organization-level impediments to GenAI red teaming and deeply integrate user research across the entire red teaming lifecycle. The work introduces a novel “user-centered red teaming framework” accompanied by targeted organizational interventions, emphasizing proactive risk identification and cross-functional collaboration. Empirical results demonstrate that this framework significantly improves red team integration within development workflows and accelerates risk response timelines. The proposed approach provides a pragmatic, organizationally grounded solution for strengthening GenAI security governance. (149 words)

Technology Category

• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

The rapid integration of generative artificial intelligence (GenAI) across diverse fields underscores the critical need for red teaming efforts to proactively identify and mitigate associated risks. While previous research primarily addresses technical aspects, this paper highlights organizational factors that hinder the effectiveness of red teaming in real-world settings. Through qualitative analysis of 15 semi-structured interviews with red teamers from various organizations, we uncover challenges such as the marginalization of vulnerable red teamers, the invisibility of nuanced AI risks to vulnerable users until post-deployment, and a lack of user-centered red teaming approaches. These issues often arise from underlying organizational dynamics, including organizational resistance, organizational inertia, and organizational mediocracy. To mitigate these dynamics, we discuss the implications of user research for red teaming and the importance of embedding red teaming throughout the entire development cycle of GenAI systems.

Problem

• Identifies organizational barriers to effective AI red teaming
• Examines marginalization of vulnerable red teamers in organizations
• Addresses lack of user-centered approaches in AI risk assessment

Innovation

• Qualitative analysis of organizational dynamics challenges
• Embedding red teaming throughout entire development cycle
• User-centered approaches to address organizational resistance