A General Framework for Error-controlled Unstructured Scientific Data Compression

2024-09-16

IEEE International Conference on e-Science

AI Summary

To address insufficient neighbor information exploitation and uncontrollable error in lossy compression of unstructured mesh scientific data, this paper proposes an error-bounded multi-component compression framework. First, irregular mesh data are interpolated onto a regular grid; then, the interpolated field and residual are compressed separately. This is the first method to achieve generic, strictly error-bounded compression for arbitrary-topology unstructured meshes. By decoupling interpolation from residual encoding, the framework seamlessly integrates with mainstream floating-point compressors such as ZFP and FPZIP. Evaluated on four representative scientific datasets, it achieves 2.3–3.5× higher average compression ratios than state-of-the-art methods while rigorously satisfying absolute error bounds ranging from 1e−6 to 1e−2—demonstrating superior accuracy-compression trade-offs.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Machine Learning: Learning on the Edge & Model Compression
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Web Mining and Content Analysis: Bridging structured and unstructured data

Abstract

Data compression plays a key role in reducing storage and I/O costs. Traditional lossy methods primarily target data on rectilinear grids and cannot leverage the spatial coherence in unstructured mesh data, leading to suboptimal compression ratios. We present a multi-component, error-bounded compression framework designed to enhance the compression of floating-point unstructured mesh data, which is common in scientific applications. Our approach involves interpolating mesh data onto a rectilinear grid and then separately compressing the grid interpolation and the interpolation residuals. This method is general, independent of mesh types and typologies, and can be seamlessly integrated with existing lossy compressors for improved performance. We evaluated our framework across twelve variables from two synthetic datasets and two real-world simulation datasets. The results indicate that the multi-component framework consistently outperforms state-of-the-art lossy compressors on unstructured data, achieving, on average, a 2.3 − 3.5× improvement in compression ratios, with error bounds ranging from 1 × 10 the −6 to 1×10−2. We further investigate impact of hyperparameters, such as grid spacing and error allocation, to deliver optimal compression ratios in diverse datasets.

Problem

• Data Compression
• Irregular Grid Data
• Error Control

Innovation

• Irregular Data Compression
• Error Control
• Square Grid Mapping