On the Costs and Benefits of Learned Indexing for Dynamic High-Dimensional Data: Extended Version

2025-07-08

AI Summary

Static learned indexes struggle to adapt to continuous data growth in high-dimensional dynamic settings. Method: This paper proposes a novel dynamic extension of static learned indexes, introducing an incremental update mechanism based on node splitting and widening to support efficient insertions and localized reconstruction; it further establishes an amortized cost model jointly optimizing index construction overhead and query latency, theoretically characterizing—for the first time—the critical conditions under which dynamic indexes outperform static ones. Contribution/Results: The work represents the first systematic adaptation of static learned indexes to high-dimensional dynamic environments, balancing accuracy, efficiency, and maintainability. Experiments demonstrate that, as data scale expands, the proposed dynamic index achieves significantly lower amortized cost than static baselines, exhibiting superior scalability and practical utility.

Technology Category

• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Intelligent Query Processing
• Machine Learning: Learning with Manifolds

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Models for Web evolution

Abstract

One of the main challenges within the growing research area of learned indexing is the lack of adaptability to dynamically expanding datasets. This paper explores the dynamization of a static learned index for complex data through operations such as node splitting and broadening, enabling efficient adaptation to new data. Furthermore, we evaluate the trade-offs between static and dynamic approaches by introducing an amortized cost model to assess query performance in tandem with the build costs of the index structure, enabling experimental determination of when a dynamic learned index outperforms its static counterpart. We apply the dynamization method to a static learned index and demonstrate that its superior scaling quickly surpasses the static implementation in terms of overall costs as the database grows. This is an extended version of the paper presented at DAWAK 2025.

Problem

• Adapting learned indexing to dynamic high-dimensional datasets
• Evaluating trade-offs between static and dynamic indexing approaches
• Demonstrating superior scaling of dynamic learned indexes

Innovation

• Dynamic learned index with node splitting
• Amortized cost model for performance trade-offs
• Superior scaling for growing databases