Can Deep Neural Networks Improve Compression of Very Large Scientific Data?

2026-06-12

AI Summary

This study investigates whether deep neural networks can enhance compression efficiency for large-scale scientific data under error-bounded constraints. For the first time, we systematically evaluate the effectiveness of pretrained foundation models in meteorology—such as GraphCast, Aurora, and VAEformer—integrated into the SZ3.1 compression framework to replace autoregressive prediction and thereby avoid error accumulation. Experimental results show that while highly predictable variables achieve up to a 9.6× compression ratio with a 91% improvement in reconstruction quality, the overall dataset compression ratio does not significantly improve. The findings reveal that prediction accuracy alone is insufficient to determine compression performance; instead, the spatial structure of prediction residuals critically influences entropy coding efficiency.

Abstract

Error-bounded lossy compression is a fundamental technique for managing the rapidly growing volumes of scientific data produced by modern simulations and observational instruments. Most state-of-the-art-compressors follow a prediction-residual paradigm, where compression effectiveness depends on the quality of the predictor: more accurate predictions generate smaller residuals that are easier to compress. This observation raises a question: can modern machine learning models serve as superior predictors for scientific data compression? Answering this question directly is challenging because developing compression-specific ML predictors requires substantial resources. Instead, we leverage the climate domain where highly accurate pretrained weather forecasting foundation models already exist, making them an ideal testbed. We present a framework that integrates spatial and temporal deep learning models into a conventional error-bounded compression pipeline. The framework supports auto-regressive forecasting models and avoids error accumulation. Using ERA5 climate data as a representative large-scale scientific dataset, we evaluate three distinct ML predictors: a VAEformer-based codec (CRA5), a graph neural network forecaster (GraphCast), and a vision-transformer forecaster (Aurora), against the state-of-the-art compressor SZ3.1 under identical quantization and entropy-coding backends. Our evaluation over approximately 1.7 TB of data reveals a surprising result: although ML predictors generate more accurate predictions and can improve reconstruction quality by up to 91% while achieving up to 9.6x higher compression ratios for highly predictable variables, they do not improve overall dataset-level compression ratio. We show that prediction accuracy alone is insufficient: the spatial structure of the resulting residuals plays a decisive role in entropy coding efficiency.

Problem

• lossy compression
• scientific data
• deep neural networks
• error-bounded compression
• compression ratio

Innovation

• error-bounded lossy compression
• deep learning predictors
• scientific data compression
• residual spatial structure
• foundation models