Image-GS: Content-Adaptive Image Representation via 2D Gaussians

2024-07-02

arXiv.org

AI Summary

Existing neural image representation methods are constrained by fixed data structures or computationally intensive implicit models, struggling to simultaneously achieve high-fidelity reconstruction, low memory footprint, and real-time rendering. This paper proposes a content-adaptive, anisotropic 2D colored Gaussian ellipse representation framework. Leveraging a differentiable renderer, it enables feature-driven dynamic sparsity modeling and error-guided progressive optimization. The representation is explicit, sparse, and hardware-friendly, supporting pixel-level random access with only 0.3K MACs per pixel. It inherently supports smooth level-of-detail (LOD) hierarchies. Extensive evaluation demonstrates significant superiority over conventional compression and implicit representations—particularly on stylized images with non-uniform textures and under low-bitrate conditions. The framework is validated across diverse tasks, including texture compression, semantics-aware compression, and joint compression-restoration, confirming its effectiveness and generalizability.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Representation Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Neural image representations have emerged as a promising approach for encoding and rendering visual data. Combined with learning-based workflows, they demonstrate impressive trade-offs between visual fidelity and memory footprint. Existing methods in this domain, however, often rely on fixed data structures that suboptimally allocate memory or compute-intensive implicit models, hindering their practicality for real-time graphics applications. Inspired by recent advancements in radiance field rendering, we introduce Image-GS, a content-adaptive image representation based on 2D Gaussians. Leveraging a custom differentiable renderer, Image-GS reconstructs images by adaptively allocating and progressively optimizing a group of anisotropic, colored 2D Gaussians. It achieves a favorable balance between visual fidelity and memory efficiency across a variety of stylized images frequently seen in graphics workflows, especially for those showing non-uniformly distributed features and in low-bitrate regimes. Moreover, it supports hardware-friendly rapid random access for real-time usage, requiring only 0.3K MACs to decode a pixel. Through error-guided progressive optimization, Image-GS naturally constructs a smooth level-of-detail hierarchy. We demonstrate its versatility with several applications, including texture compression, semantics-aware compression, and joint image compression and restoration.

Problem

• Optimizing memory and compute for neural image representations
• Balancing visual fidelity and efficiency in graphics workflows
• Enabling real-time usage with hardware-friendly rapid access

Innovation

• Uses 2D Gaussians for adaptive image representation
• Leverages differentiable renderer for optimization
• Supports real-time usage with low MACs