NeurDB: On the Design and Implementation of an AI-powered Autonomous Database

2024-08-06

arXiv.org

AI Summary

Existing AI-powered database systems struggle to adapt to dynamically shifting data distributions and workloads in real-world deployments, resulting in poor adaptability and suboptimal, lagging performance tuning. This paper introduces the first AI-driven autonomous database system capable of self-adapting to both data and workload drift, achieving end-to-end intelligent optimization via deep kernel-level AI integration. Our key contributions are: (1) the first in-database AI ecosystem designed for dynamic evolution; and (2) a suite of fast-adaptive learning components—including learned indexes, a self-tuning query optimizer, workload-aware model update mechanisms, and a unified AI-native execution engine. Under realistic workloads, our system improves AI analytics throughput by 3.2× and reduces latency by 67%. In drift-prone scenarios, it demonstrates significantly superior performance stability compared to state-of-the-art approaches.

Technology Category

• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Intelligent Query Processing
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Databases are increasingly embracing AI to provide autonomous system optimization and intelligent in-database analytics, aiming to relieve end-user burdens across various industry sectors. Nonetheless, most existing approaches fail to account for the dynamic nature of databases, which renders them ineffective for real-world applications characterized by evolving data and workloads. This paper introduces NeurDB, an AI-powered autonomous database that deepens the fusion of AI and databases with adaptability to data and workload drift. NeurDB establishes a new in-database AI ecosystem that seamlessly integrates AI workflows within the database. This integration enables efficient and effective in-database AI analytics and fast-adaptive learned system components. Empirical evaluations demonstrate that NeurDB substantially outperforms existing solutions in managing AI analytics tasks, with the proposed learned components more effectively handling environmental dynamism than state-of-the-art approaches.

Problem

• Database Systems
• Artificial Intelligence
• Adaptability

Innovation

• AI-controlled
• intelligent database system
• environmental changes responsiveness