NMP-PaK: Near-Memory Processing Acceleration of Scalable De Novo Genome Assembly

2025-05-12

AI Summary

To address computational bottlenecks in *de novo* genome assembly—stemming from massive data volumes, irregular memory access patterns, and high memory bandwidth demands—this work proposes a hardware-software co-design for near-memory processing (NMP). We introduce a channel-level NMP architecture integrating specialized processing units for dynamic, interdependent data structures, and design a CPU-NMP hybrid execution model coupled with memory-aware batching to enable efficient parallel processing of graph-structured assembly data. Experimental evaluation demonstrates that, compared to state-of-the-art tools, our approach reduces memory footprint by 14×, improves end-to-end performance by 16×, decreases memory operations by 2.4×, and achieves 8.3× higher throughput under identical resource constraints—significantly alleviating memory bandwidth saturation and poor memory access locality.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Learning & Optimization for NLP
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

De novo assembly enables investigations of unknown genomes, paving the way for personalized medicine and disease management. However, it faces immense computational challenges arising from the excessive data volumes and algorithmic complexity. While state-of-the-art de novo assemblers utilize distributed systems for extreme-scale genome assembly, they demand substantial computational and memory resources. They also fail to address the inherent challenges of de novo assembly, including a large memory footprint, memory-bound behavior, and irregular data patterns stemming from complex, interdependent data structures. Given these challenges, de novo assembly merits a custom hardware solution, though existing approaches have not fully addressed the limitations. We propose NMP-PaK, a hardware-software co-design that accelerates scalable de novo genome assembly through near-memory processing (NMP). Our channel-level NMP architecture addresses memory bottlenecks while providing sufficient scratchpad space for processing elements. Customized processing elements maximize parallelism while efficiently handling large data structures that are both dynamic and interdependent. Software optimizations include customized batch processing to reduce the memory footprint and hybrid CPU-NMP processing to address hardware underutilization caused by irregular data patterns. NMP-PaK conducts the same genome assembly while incurring a 14X smaller memory footprint compared to the state-of-the-art de novo assembly. Moreover, NMP-PaK delivers a 16X performance improvement over the CPU baseline, with a 2.4X reduction in memory operations. Consequently, NMP-PaK achieves 8.3X greater throughput than state-of-the-art de novo assembly under the same resource constraints, showcasing its superior computational efficiency.

Problem

• Accelerating scalable de novo genome assembly
• Reducing memory footprint and bottlenecks
• Handling dynamic, interdependent data structures efficiently

Innovation

• Near-memory processing for genome assembly acceleration
• Customized processing elements for dynamic data structures
• Hybrid CPU-NMP processing to optimize resource utilization