On Usage of Non-Volatile Memory as Primary Storage for Database Management Systems

2025-02-13

AI Summary

This work addresses the high memory access latency and performance bottlenecks arising from data-computation separation in NVM-main-memory database systems. To tackle these challenges, we propose a co-optimization architecture comprising two synergistic components: (1) a deep redesign of the PostgreSQL storage engine to fully exploit NVM’s byte-addressability, low latency, and persistence; and (2) a lightweight, general-purpose asynchronous prefetching library that mitigates NVM access bottlenecks via multi-threaded prefetching and low-overhead API abstractions. Experimental evaluation shows that the engine redesign alone reduces query latency by 19% (vs. disk baseline) and 4% (vs. NVM baseline) on average; integrating the prefetching library further improves reductions to 23% and 8%, respectively—with peak speedups reaching 54% over disk. This work pioneers the “engine refactoring + system-level prefetching” co-design paradigm, delivering a reusable architectural blueprint and practical implementation pathway for NVM-native DBMS development.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Hardware-aware ML

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

This paper explores the implications of employing non-volatile memory (NVM) as primary storage for a data base management system (DBMS). We investigate the modifications necessary to be applied on top of a traditional relational DBMS to take advantage of NVM features. As a case study, we modify the storage engine (SE) of PostgreSQL enabling efficient use of NVM hardware. We detail the necessary changes and challenges such modifications entail and evaluate them using a comprehensive emulation platform. Results indicate that our modified SE reduces query execution time by up to 45% and 13% when compared to disk and NVM storage, with average reductions of 19% and 4%, respectively. Detailed analysis of these results shows that while our modified SE is able to access data more efficiently, data is not close to the processing units when needed for processing, incurring long latency misses that hinder the performance. To solve this, we develop a general purpose library that employs helper threads to prefetch data from NVM hardware via a simple API. Our library further improves query execution time for our modified SE when compared to disk and NVM storage by up to 54% and 17%, with average reductions of 23% and 8%, respectively.

Problem

• Explores NVM as primary DBMS storage
• Modifies PostgreSQL for NVM efficiency
• Develops library to reduce query latency

Innovation

• Modified PostgreSQL for NVM
• Developed prefetch library API
• Enhanced query execution performance