FZModules: A Heterogeneous Computing Framework for Customizable Scientific Data Compression Pipelines

2025-09-24

AI Summary

Scientific data explosion has exacerbated storage and memory bottlenecks, while existing lossy compression schemes struggle to jointly optimize rate-distortion performance, throughput, and customizability: CPU-based methods achieve high fidelity but suffer from low throughput; GPU kernel-fused accelerators deliver high throughput but lack flexibility and exhibit poor rate-distortion trade-offs; expert-level tuning remains prohibitively complex. This paper introduces the first modular heterogeneous compression framework, enabling flexible composition of CPU and GPU components via a concise, unified interface. It integrates asynchronous task scheduling, dynamic data-dependency inference, and zero-copy memory management to support fine-grained (branch- and stage-level) concurrency and hybrid CPU-GPU execution. Evaluated on four representative scientific datasets, our framework achieves end-to-end throughput comparable to state-of-the-art GPU-fused kernels, while significantly outperforming existing GPU-based approaches in rate-distortion performance—and matching or exceeding high-fidelity CPU and CPU-GPU hybrid methods.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Learning & Optimization for CV

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Modern scientific simulations and instruments generate data volumes that overwhelm memory and storage, throttling scalability. Lossy compression mitigates this by trading controlled error for reduced footprint and throughput gains, yet optimal pipelines are highly data and objective specific, demanding compression expertise. GPU compressors supply raw throughput but often hard-code fused kernels that hinder rapid experimentation, and underperform in rate-distortion. We present FZModules, a heterogeneous framework for assembling error-bounded custom compression pipelines from high-performance modules through a concise extensible interface. We further utilize an asynchronous task-backed execution library that infers data dependencies, manages memory movement, and exposes branch and stage level concurrency for powerful asynchronous compression pipelines. Evaluating three pipelines built with FZModules on four representative scientific datasets, we show they can compare end-to-end speedup of fused-kernel GPU compressors while achieving similar rate-distortion to higher fidelity CPU or hybrid compressors, enabling rapid, domain-tailored design.

Problem

• Addresses overwhelming data volumes from scientific simulations that strain memory and storage systems
• Solves rigid GPU compressor designs that hinder rapid experimentation and underperform in rate-distortion
• Enables customizable compression pipelines balancing speed with quality for scientific data

Innovation

• Heterogeneous framework for customizable compression pipelines
• Asynchronous task execution with dependency inference
• Modular design balancing GPU speed and CPU fidelity