Can AI autonomously build, operate, and use the entire data stack?

2025-12-08

AI Summary

Current AI systems only assist specific roles in isolated data management tasks, failing to achieve end-to-end autonomous governance across the enterprise data stack lifecycle. Method: This paper proposes a novel paradigm of autonomous data systems based on multi-agent collaboration, integrating intelligent agents, autonomous decision-making, AI-driven governance, and quality optimization—embedded within modern data stack architectures to enable dynamic, closed-loop data architecture design, integration, monitoring, and evolution. Contribution/Results: The study demonstrates the feasibility of fully autonomous data systems, identifies critical challenges—including cross-layer semantic alignment, trustworthy autonomy boundaries, and human-AI co-governance—and establishes a scalable, auditable, human-AI shared, self-sustaining data ecosystem framework. These contributions provide a foundational methodology and technical roadmap for next-generation highly autonomous data infrastructure.

Technology Category

• Multiagent Systems: Agent/AI Theories and Architectures
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Security and Privacy: Data transparency and provenance

Abstract

Enterprise data management is a monumental task. It spans data architecture and systems, integration, quality, governance, and continuous improvement. While AI assistants can help specific persona, such as data engineers and stewards, to navigate and configure the data stack, they fall far short of full automation. However, as AI becomes increasingly capable of tackling tasks that have previously resisted automation due to inherent complexities, we believe there is an imminent opportunity to target fully autonomous data estates. Currently, AI is used in different parts of the data stack, but in this paper, we argue for a paradigm shift from the use of AI in independent data component operations towards a more holistic and autonomous handling of the entire data lifecycle. Towards that end, we explore how each stage of the modern data stack can be autonomously managed by intelligent agents to build self-sufficient systems that can be used not only by human end-users, but also by AI itself. We begin by describing the mounting forces and opportunities that demand this paradigm shift, examine how agents can streamline the data lifecycle, and highlight open questions and areas where additional research is needed. We hope this work will inspire lively debate, stimulate further research, motivate collaborative approaches, and facilitate a more autonomous future for data systems.

Problem

• AI autonomously manages entire data lifecycle
• Shifts from component AI to holistic automation
• Builds self-sufficient systems for humans and AI

Innovation

• AI autonomously manages entire data lifecycle
• Shift from component AI to holistic autonomous systems
• Intelligent agents build self-sufficient data systems