The Perils of Agency: How Developers Perceive, Prioritize, and Address Risks in Agentic AI Products

2026-06-13

AI Summary

While autonomous agents enhance autonomy and environmental adaptability, they simultaneously amplify product risks, yet current development practices lack effective means to manage these risks without compromising the agents’ core capabilities. Through semi-structured interviews with 35 industry practitioners followed by thematic analysis, this study systematically uncovers, for the first time, how developers prioritize risks—typically emphasizing product and business concerns while overlooking societal implications—and often impose constraints that inadvertently degrade the agents’ essential functionalities. The work introduces the “capability–risk control” paradox, articulating the inherent tension between the intrinsic characteristics of autonomous agents and the demands of risk mitigation. This insight provides a theoretical foundation and practical guidance for designing autonomous agents that effectively balance performance with safety.

Abstract

Agentic AI systems act autonomously, use tools, adapt to context, and operate in complex real-world environments. However, these same characteristics can create or exacerbate product risks. We studied how industry developers (n=35) perceive, prioritize, and address the risks in their agentic AI products. We found that developers' perceptions of risk were closely tied to the qualities that made the product agentic, such as autonomy, tool use, and usage in a real-world context. Developers prioritized product and business risks before considering downstream societal risks like job displacement and end-user privacy. This prioritization also impacted developers' ability and motivation to mitigate agentic risks. Finally, developers lacked mature controls for containing agentic risks, often relying on constraining the same characteristics that make agents useful: e.g., autonomy and goal complexity. These findings reveal a capability vs. risk control tension in agentic AI development: developers need to address risks that emerge from agentic capabilities, yet they currently have limited support for doing so without constraining agentic functionality.

Problem

• Agentic AI
• risk perception
• autonomy
• AI safety
• developer practices

Innovation

• agentic AI
• risk perception
• autonomy
• tool use
• risk mitigation