🤖 AI Summary
This work addresses the frequent disruptions in data and AI pipelines caused by data anomalies, schema changes, or infrastructure failures, which existing self-healing solutions often mitigate at high cost, with vendor lock-in, or with limited adaptability for small-to-medium teams. The paper proposes an open-source, vendor-agnostic autonomous remediation architecture that integrates monitoring, metadata, historical incident logs, a policy engine, AI-driven root cause analysis, and controlled repair mechanisms to automatically detect, diagnose, remediate, and validate pipeline issues. Its key contribution lies in delivering a unified, portable end-to-end reference architecture that systematically consolidates fragmented capabilities without reliance on proprietary platforms, substantially reducing manual intervention and enhancing pipeline resilience across diverse contexts such as data engineering, MLOps, and software delivery.
📝 Abstract
Modern organizations rely on data, machine learning, and software delivery pipelines to move data, train models, deploy applications, refresh dashboards, and support business-critical decisions. However, these pipelines often fail because of data quality issues, schema changes, upstream source changes, infrastructure problems, orchestration failures, and model workflow issues. Existing ZeroOps, observability, and AI operations platforms can help teams detect incidents, investigate root causes, and in some cases recommend or execute fixes. However, many of these solutions are expensive, vendor-specific, or difficult for smaller teams to adapt across different tools and environments. This paper first compares existing off-the-shelf solutions for AI-assisted pipeline monitoring, root-cause analysis, and automated remediation, including their strengths, limitations, and practical trade-offs. Based on this comparison, we find that the main gap is architectural rather than technological: the required ingredients for self-healing pipelines already exist, but they are fragmented across vendor-specific platforms, observability tools, incident systems, and open-source components. We therefore propose an affordable, vendor-agnostic reference architecture for agentic self-healing pipelines using open-source and low-cost tools. The proposed architecture combines monitoring, pipeline metadata, incident history, deterministic policy checks, AI-assisted diagnosis, approval workflows, and controlled remediation actions to help teams detect, diagnose, repair, verify, and learn from pipeline issues with less manual effort. The goal is to provide a practical reference architecture that can be adapted across data engineering, machine learning operations, and software delivery environments.