Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings

2025-04-21

AI Summary

Existing columnar storage formats (e.g., Parquet, Arrow) suffer from severely limited random-access performance on NVMe devices, failing to simultaneously achieve low-latency random reads and high-throughput full scans. To address this, we propose a width-aware dual-mode structural encoding scheme: it dynamically selects between sparse and dense encoding strategies based on per-column width, while jointly modeling NVMe I/O characteristics and optimizing columnar data layout to co-optimize random access and sequential scan performance. Implemented in the Lance storage engine, our approach achieves a 60× improvement in random-access throughput over default Parquet, with near-identical full-scan throughput and memory footprint. It significantly outperforms both Arrow and Parquet baselines across key metrics, demonstrating superior trade-off balancing between latency-sensitive random access and bandwidth-intensive scanning workloads on modern NVMe storage.

Technology Category

• Search and Optimization: Distributed Search
• Machine Learning: Hardware-aware ML
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

The growing interest in artificial intelligence has created workloads that require both sequential and random access. At the same time, NVMe-backed storage solutions have emerged, providing caching capability for large columnar datasets in cloud storage. Current columnar storage libraries fall short of effectively utilizing an NVMe device's capabilities, especially when it comes to random access. Historically, this has been assumed an implicit weakness in columnar storage formats, but this has not been sufficiently explored. In this paper, we examine the effectiveness of popular columnar formats such as Apache Arrow, Apache Parquet, and Lance in both random access and full scan tasks against NVMe storage. We argue that effective encoding of a column's structure, such as the repetition and validity information, is the key to unlocking the disk's performance. We show that Parquet, when configured correctly, can achieve over 60x better random access performance than default settings. We also show that this high random access performance requires making minor trade-offs in scan performance and RAM utilization. We then describe the Lance structural encoding scheme, which alternates between two different structural encodings based on data width, and achieves better random access performance without making trade-offs in scan performance or RAM utilization.

Problem

• Improving random access in columnar storage with NVMe
• Optimizing structural encodings for better disk performance
• Balancing random access and scan performance trade-offs

Innovation

• Adaptive structural encodings for columnar storage
• Optimized NVMe performance for random access
• Balanced trade-offs in scan and RAM usage