STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data

2025-09-01

AI Summary

Addressing the challenge of simultaneously achieving progressive decompression, random access, and high compression quality/speed in lossy scientific data compression, this paper proposes the first unified framework supporting both decompression modes. Our method employs hierarchical data partitioning and hierarchical prediction, integrated with error-bounded lossy compression and streaming-based codec design. It maintains reconstruction accuracy comparable to state-of-the-art non-streaming compressors (e.g., SZ3), while significantly improving decompression throughput—up to 6.7× faster than SZ3. To our knowledge, this is the first approach to jointly optimize high fidelity, high throughput, and flexible data access. The framework enables practical online analysis and on-demand data retrieval, bridging a critical gap between theoretical compression efficiency and real-world scientific computing workflows.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Machine Learning: Time-Series/Data Streams
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality -- even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7$ imes$ faster than SZ3.

Problem

• Enabling progressive and random-access decompression without quality loss
• Maintaining high compression speed while supporting streaming features
• Overcoming partitioning impact to achieve state-of-the-art compression quality

Innovation

• Streaming compression with progressive and random-access decompression
• Hierarchical partitioning strategy for streaming features
• High compression speed and quality comparable to SZ3