Builder, Defender, Breaker: The Case Against Removing the Human from the AI-Driven Security Lifecycle

2026-07-03

AI Summary

This work addresses the multifaceted role of AI systems in the security lifecycle—as builders, defenders, and attackers—and the resulting systemic blind spots, validation failures, lack of human intervention, and accountability challenges. It proposes a new paradigm centered on “humans as a permanent structural element,” arguing that human–AI collaborative security frameworks must preserve human autonomy, intervenability, and clear accountability mechanisms. Through an integrated empirical approach combining autonomous code generation, adversarial machine learning, software fault tolerance, and fully automated red–blue teaming, the study exposes the inherent fragility of end-to-end automated security pipelines. These findings establish a theoretical foundation and design principles for developing trustworthy, auditable, and adversarially robust human–AI security architectures.

Abstract

Artificial intelligence has spread across the whole of the security lifecycle. The same family of models now writes application code, hardens it, and probes it for weaknesses, so that a single generative substrate increasingly performs all three roles at once. Enthusiasm for this convergence tends to treat full autonomy as the natural end point of partial assistance. This article argues that it is not. When the system that builds an artifact is drawn from the same distribution as the systems that defend and test it, the three roles inherit a common set of blind spots, and the independence that makes verification meaningful is quietly lost. Removing the human does more than raise the automation level: it collapses the external oracle against which machine output is judged, outruns the point at which a person could intervene, hands adversaries a predictable and poisonable target, and dissolves the locus of accountability when something fails. Drawing on evidence from autonomous code generation, adversarial machine learning, software fault tolerance, and the first all-machine hacking tournaments, we argue that the human belongs in the loop not as a temporary scaffold but as a permanent structural requirement, and set out what a defensible division of labour between people and machines should preserve.

Problem

• AI-driven security lifecycle
• human-in-the-loop
• autonomous systems
• adversarial machine learning
• accountability

Innovation

• human-in-the-loop
• AI security lifecycle
• role convergence
• adversarial machine learning
• accountability