BaCon: Efficient Batch Processing of Counting Queries [Full Version]

2026-07-07

AI Summary

This work addresses the high cost of batch-processing large-scale cardinality queries, which severely hinders the training and maintenance of learned cardinality estimation models. To overcome this challenge, the paper proposes BaCon, a client-side, kernel-agnostic batching algorithm that achieves high efficiency without requiring modifications to the database engine. BaCon’s key innovation lies in its novel integration of factorized databases with a workload-aware domain quantization strategy, which avoids materializing expensive join results while substantially reducing data generation overhead. Implemented as a lightweight C-language user-defined function compatible with PostgreSQL clients, BaCon demonstrates significant performance gains, achieving speedups of 2× to 178× over baseline methods across diverse workloads, thereby greatly enhancing the training efficiency and practicality of learned cardinality estimators.

Abstract

Counting queries are ubiquitous in database systems, particularly for driving internal system optimization. Learned models for cardinality estimation rely heavily on large-scale training data, yet generating such data by executing massive batches of counting queries is expensive. We propose BaCon, an efficient algorithm for batch evaluation of counting queries on top of a database system, without modifying its internals. BaCon integrates the idea of factorized databases with a workload-aware domain quantization strategy, allowing it to evaluate batches of counting queries using compact data structures rather than materializing massive join results. BaCon's design is compatible with most database management system, and we have implemented it as a client-side application on PostgreSQL with a lightweight C-language UDF (user-defined function). This implementation delivers speedups between 2$\times$ and 178$\times$ over baselines and good performance across various workloads, making training and maintenance of learned cardinality estimation models significantly more practical.

Problem

• counting queries
• cardinality estimation
• batch processing
• database systems
• training data generation

Innovation

• factorized databases
• workload-aware domain quantization
• batch counting queries
• cardinality estimation
• client-side optimization