PUL: Pre-load in Software for Caches Wouldn't Always Play Along

2025-06-20

AI Summary

Memory latency and bandwidth bottlenecks continue to impede system performance in the post-Moore era. To address this, we present the first systematic demonstration that software prefetching—under near-data processing (NDP) architectures—exhibits superior scalability and efficiency over conventional hardware prefetching. We propose a lightweight preloading paradigm tailored for intelligent memory, which jointly optimizes latency hiding and computational resource utilization via software-driven interleaving of computation and I/O scheduling, coupled with memory-bandwidth-aware load preloading. Experimental evaluation shows that our approach significantly improves compute-unit utilization, and its prefetching efficiency consistently increases with CPU process node advancements. Across multiple generations of hardware platforms, it achieves superior latency-hiding effectiveness compared to state-of-the-art hardware prefetchers.

Technology Category

• Machine Learning: Hardware-aware ML
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Memory latencies and bandwidth are major factors, limiting system performance and scalability. Modern CPUs aim at hiding latencies by employing large caches, out-of-order execution, or complex hardware prefetchers. However, software-based prefetching exhibits higher efficiency, improving with newer CPU generations. In this paper we investigate software-based, post-Moore systems that offload operations to intelligent memories. We show that software-based prefetching has even higher potential in near-data processing settings by maximizing compute utilization through compute/IO interleaving.

Problem

• Address memory latency and bandwidth limitations in system performance
• Explore software-based prefetching efficiency in post-Moore systems
• Optimize compute utilization via compute/IO interleaving in near-data processing

Innovation

• Software-based prefetching for cache efficiency
• Offloading operations to intelligent memories
• Compute/IO interleaving for near-data processing