OnPair: Short Strings Compression for Fast Random Access

2025-08-04

AI Summary

Memory-resident databases require compression schemes that simultaneously achieve high compression ratios and low-latency random access—yet existing methods (e.g., BPE, FSST) struggle to balance both. This paper introduces OnPair, a cache-friendly dictionary-based compression algorithm tailored for in-memory databases. Its core innovation is a single-pass, incremental substring merging mechanism that constructs a dictionary of frequent adjacent substrings without global position tracking; the OnPair16 variant further constrains dictionary entries to 16 bytes and employs optimized longest-prefix matching for efficient parsing. Experiments on real-world datasets show that OnPair matches BPE’s compression ratio while accelerating compression by up to 3.2× and reducing memory overhead by 40–65%. It significantly outperforms state-of-the-art alternatives, achieving superior trade-offs among compression efficiency, random-access performance, and resource consumption.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Distributed Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

We present OnPair, a dictionary-based compression algorithm designed to meet the needs of in-memory database systems that require both high compression and fast random access. Existing methods either achieve strong compression ratios at significant computational and memory cost (e.g., BPE) or prioritize speed at the expense of compression quality (e.g., FSST). OnPair bridges this gap by employing a cache-friendly dictionary construction technique that incrementally merges frequent adjacent substrings in a single sequential pass over a data sample. This enables fast, memory-efficient training without tracking global pair positions, as required by traditional BPE. We also introduce OnPair16, a variant that limits dictionary entries to 16 bytes, enabling faster parsing via optimized longest prefix matching. Both variants compress strings independently, supporting fine-grained random access without block-level overhead. Experiments on real-world datasets show that OnPair and OnPair16 achieve compression ratios comparable to BPE while significantly improving compression speed and memory usage.

Problem

• Bridges gap between high compression and fast random access
• Reduces computational and memory costs for string compression
• Enables fine-grained random access without block-level overhead

Innovation

• Cache-friendly dictionary construction technique
• Incremental merging of frequent adjacent substrings
• Optimized longest prefix matching for fast parsing