Blocked Bloom Filters with Choices

2025-01-31

AI Summary

Existing Blocked Bloom Filters suffer from an inherent trade-off between storage efficiency and false positive rate. This paper proposes the Multi-Block Selection Blocked Bloom Filter (MB-BF), the first to integrate multi-choice hashing into the Blocked Bloom architecture: each key dynamically selects its optimal storage block among multiple candidates, thereby jointly optimizing online insertion, overload robustness, and low memory overhead. MB-BF achieves this through block partitioning, multi-hash indexing, compact bitmap encoding, and memory-layout optimization—preserving both batch query performance and dynamic update capability while overcoming traditional spatial bottlenecks. Experimental results show that, at equal false positive rates, MB-BF reduces memory usage by 15–30%; conversely, at equal memory budgets, it lowers the false positive rate by an order of magnitude. MB-BF has been validated on real-world bioinformatics tasks, including genomic sequence indexing.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Search and Optimization: Distributed Search
• Machine Learning: Hardware-aware ML

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Security and Privacy: Blockchains and distributed ledgers
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Probabilistic filters are approximate set membership data structures that represent a set of keys in small space, and answer set membership queries without false negative answers, but with a certain allowed false positive probability. Such filters are widely used in database systems, networks, storage systems and in biological sequence analysis because of their fast query times and low space requirements. Starting with Bloom filters in the 1970s, many filter data structures have been developed, each with its own advantages and disadvantages, e.g., Blocked Bloom filters, Cuckoo filters, XOR filters, Ribbon filters, and more. We introduce Blocked Bloom filters with choices that work similarly to Blocked Bloom filters, except that for each key there are two (or more) alternative choices of blocks where the key's information may be stored. The result is a filter that partially inherits the advantages of a Blocked Bloom filter, such as the ability to insert keys rapidly online or the ability to slightly overload the filter with only a small penalty to the false positive rate. At the same time, it avoids the major disadvantage of a Blocked Bloom filter, namely the larger space consumption. Our new data structure uses less space at the same false positive rate, or has a lower false positive rate at the same space consumption as a Blocked Bloom filter. We discuss the methodology, engineered implementation, a detailed performance evaluation and use cases in bioinformatics of Blocked Bloom filters with choices, showing that they can be of practical value. The implementation of the evaluated filters and the workflows used are provided via Gitlab at https://gitlab.com/rahmannlab/blowchoc-filters.

Problem

• Blocked Bloom Filters
• Storage Efficiency
• False Positive Rate

Innovation

• Blocked Bloom Filters with Choices
• Reduced False Positive Rate
• Bioinformatics Big Data Processing