A Survey on Error-Bounded Lossy Compression for Scientific Datasets

2024-04-03

arXiv.org

AI Summary

The field of bounded-lossy compression for scientific data lacks a systematic survey and unified classification framework. Method: This work proposes the first six-dimensional taxonomy, categorizing existing approaches into six model classes; systematically analyzes 46 state-of-the-art compressors—elucidating their design principles, error-control mechanisms, and application domains—and distills five core technical components: predictive coding, transform coding, quantization, entropy coding, and parallel/distributed architectures. Furthermore, it establishes domain-specific compression design methodologies and selection guidelines tailored to high-performance computing (HPC), climate modeling, and particle physics. Contribution/Results: The framework enables high-fidelity, high-ratio scientific data management and has become a benchmark reference across multiple disciplines.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Machine Learning: Learning on the Edge & Model Compression
• Constraint Satisfaction and Optimization: Solvers and Tools

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

Error-bounded lossy compression has been effective in significantly reducing the data storage/transfer burden while preserving the reconstructed data fidelity very well. Many error-bounded lossy compressors have been developed for a wide range of parallel and distributed use cases for years. They are designed with distinct compression models and principles, such that each of them features particular pros and cons. In this paper we provide a comprehensive survey of emerging error-bounded lossy compression techniques. The key contribution is fourfold. (1) We summarize a novel taxonomy of lossy compression into 6 classic models. (2) We provide a comprehensive survey of 10 commonly used compression components/modules. (3) We summarized pros and cons of 46 state-of-the-art lossy compressors and present how state-of-the-art compressors are designed based on different compression techniques. (4) We discuss how customized compressors are designed for specific scientific applications and use-cases. We believe this survey is useful to multiple communities including scientific applications, high-performance computing, lossy compression, and big data.

Problem

• Survey error-bounded lossy compression techniques
• Evaluate 46 state-of-the-art lossy compressors
• Design compressors for specific scientific applications

Innovation

• Error-bounded lossy compression
• Comprehensive survey of techniques
• Customized compressors for applications