🤖 AI Summary
This study addresses the systemic risks posed by on-premises AI coding agents, whose autonomous modifications to code and infrastructure may lead to severe organizational or societal harm due to challenges in timely constraint, auditing, or reversal. For the first time, it systematically applies three systems safety methodologies—STECA, STPA, and FRAM—to model risks in cutting-edge laboratory settings from multiple perspectives. The analysis reveals critical blind spots in current AI governance frameworks, particularly concerning unverifiable accountability, control failure caused by intervention delays, and weakened safeguards due to operational drift. The work underscores the necessity of integrating model-level evaluations with system-level hazard analysis, offering a crucial complementary pathway for robust AI risk management.
📝 Abstract
Internal deployment of agentic AI systems for coding and research creates a sociotechnical control problem that extends beyond model behaviour. We treat internal-deployment Loss of Control as the inability to reliably constrain, audit, reverse, or halt AI-mediated changes to code, infrastructure, evaluation, or deployment processes in time to prevent serious organisational or societal harms. We ask whether established systems-safety methods can identify risks that model-level evaluations may miss. Using a generic frontier-lab coding-agent scenario reconstructed from public materials, we apply STECA, STPA, and FRAM. The analyses surface complementary findings: published frameworks can leave governance responsibilities and feedback loops externally unverifiable; delays in monitoring and intervention can make otherwise valid control actions ineffective; and routine operational variability can gradually erode the calibration and independence of safeguards. We argue that frontier-AI risk management should pair model-focused evaluations with systems-level hazard analysis and operational assurance that tracks whether controls remain effective over time.