A Unified Model for Cardinality Estimation by Learning from Data and Queries via Sum-Product Networks

2025-05-13

AI Summary

Cardinality estimation in databases must balance accuracy, inference latency, and storage overhead—yet existing approaches fail to jointly optimize all three. This paper introduces Query-aware Sum-Product Networks (QSPNs), the first method to explicitly model query access patterns via novel QProduct and QSplit nodes, enabling joint optimization of query-aware column partitioning and tree-structure expansion. Integrating query workload analysis, offline learning, and online fast inference, QSPN achieves state-of-the-art performance across both single-table and multi-table settings: it attains the lowest estimation error, sub-millisecond inference latency, and only MB-scale model storage. Its core contribution is the first end-to-end, lightweight, and query-aware cardinality estimation framework that simultaneously delivers high accuracy, ultra-low latency, and minimal memory footprint.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Distributed Search

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Cardinality estimation is a fundamental component in database systems, crucial for generating efficient execution plans. Despite advancements in learning-based cardinality estimation, existing methods may struggle to simultaneously optimize the key criteria: estimation accuracy, inference time, and storage overhead, limiting their practical applicability in real-world database environments. This paper introduces QSPN, a unified model that integrates both data distribution and query workload. QSPN achieves high estimation accuracy by modeling data distribution using the simple yet effective Sum-Product Network (SPN) structure. To ensure low inference time and reduce storage overhead, QSPN further partitions columns based on query access patterns. We formalize QSPN as a tree-based structure that extends SPNs by introducing two new node types: QProduct and QSplit. This paper studies the research challenges of developing efficient algorithms for the offline construction and online computation of QSPN. We conduct extensive experiments to evaluate QSPN in both single-table and multi-table cardinality estimation settings. The experimental results have demonstrated that QSPN achieves superior and robust performance on the three key criteria, compared with state-of-the-art approaches.

Problem

• Improving cardinality estimation accuracy in database systems
• Reducing inference time and storage overhead simultaneously
• Integrating data distribution and query workload for optimization

Innovation

• Uses Sum-Product Networks for data distribution modeling
• Partitions columns based on query access patterns
• Introduces QProduct and QSplit node types