Guaranteed Conditional Diffusion: 3D Block-based Models for Scientific Data Compression

2025-02-18

AI Summary

This work addresses the challenge of achieving both high compression ratios and strict error control in lossy scientific data compression. We propose GCDTC, a conditional diffusion framework with guaranteed error bounds. Methodologically, we introduce a novel coupling of 3D block-structured modeling with 2D conditional reverse diffusion, incorporating a hybrid 3D-conditional/2D-denoising U-Net, deterministic zero-noise inversion, and a tensor-based error correction module—enabling provable upper bounds on reconstruction error. Our key contributions are: (i) the first error-verifiable conditional diffusion paradigm for scientific data compression; and (ii) state-of-the-art compression fidelity on climate and combustion simulation datasets—matching or exceeding leading conventional methods while substantially outperforming standard convolutional autoencoders—and providing rigorous theoretical guarantees on the maximum attainable reconstruction error.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Diffusion Models for Vision

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Security and Privacy: Data transparency and provenance

Abstract

This paper proposes a new compression paradigm -- Guaranteed Conditional Diffusion with Tensor Correction (GCDTC) -- for lossy scientific data compression. The framework is based on recent conditional diffusion (CD) generative models, and it consists of a conditional diffusion model, tensor correction, and error guarantee. Our diffusion model is a mixture of 3D conditioning and 2D denoising U-Net. The approach leverages a 3D block-based compressing module to address spatiotemporal correlations in structured scientific data. Then, the reverse diffusion process for 2D spatial data is conditioned on the ``slices'' of content latent variables produced by the compressing module. After training, the denoising decoder reconstructs the data with zero noise and content latent variables, and thus it is entirely deterministic. The reconstructed outputs of the CD model are further post-processed by our tensor correction and error guarantee steps to control and ensure a maximum error distortion, which is an inevitable requirement in lossy scientific data compression. Our experiments involving two datasets generated by climate and chemical combustion simulations show that our framework outperforms standard convolutional autoencoders and yields competitive compression quality with an existing scientific data compression algorithm.

Problem

• Lossy scientific data compression
• Control maximum error distortion
• Leverage 3D block-based compression

Innovation

• Guaranteed Conditional Diffusion model
• 3D block-based compressing module
• Tensor correction and error guarantee