Fully Autonomous AI Agents Should Not be Developed

2025-02-04

AI Summary

This paper examines the ethical legitimacy and associated risks of developing fully autonomous AI agents. It addresses concerns regarding uncontrolled delegation of decision-making authority and its cascading safety and ethical implications. Method: Drawing on a hierarchical taxonomy of AI agents, the study integrates perspectives from AI governance, human–AI interaction, and philosophy of technology. It employs an interdisciplinary ethical analysis framework and case studies of mainstream commercial AI agents to empirically test the hypothesis that agent autonomy correlates positively with human risk. Contribution/Results: The work establishes the “autonomy–risk positive correlation” principle, providing the first systematic, functionally modeled linkage between graded autonomy levels and quantifiable risk—explicitly embedded within a governance framework. Findings inform regulatory policy design and system architecture, advocating a paradigm shift toward supervisable, interruptible, and user-centric AI agents. This constitutes the inaugural study to formalize autonomy–risk relationships through mathematical modeling and integrate them directly into AI governance theory and practice.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Multiagent Systems: Agent/AI Theories and Architectures
• Humans and AI: Learning Human Values and Preferences

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

This paper argues that fully autonomous AI agents should not be developed. In support of this position, we build from prior scientific literature and current product marketing to delineate different AI agent levels and detail the ethical values at play in each, documenting trade-offs in potential benefits and risks. Our analysis reveals that risks to people increase with the autonomy of a system: The more control a user cedes to an AI agent, the more risks to people arise. Particularly concerning are safety risks, which affect human life and impact further values.

Problem

• Risks increase with AI autonomy
• Safety risks affect human life
• Ethical values in AI development

Innovation

• AI agent autonomy levels
• ethical value analysis
• safety risk assessment