ZoomLDM: Latent Diffusion Model for multi-scale image generation

2024-11-25

arXiv.org

AI Summary

Existing diffusion models struggle with generating ultra-high-resolution images (e.g., digital pathology and satellite imagery) due to fixed, small-scale receptive fields, limiting global structural modeling and multi-scale semantic coherence. Method: We propose the first diffusion framework for large-image, multi-scale generation. Built upon a latent diffusion architecture, it introduces three key components: (1) a zoom-aware conditional mechanism, (2) self-supervised learning (SSL)-based feature encoding for robust representation, and (3) multi-scale latent-space conditioning with progressive scale-alignment training. Contribution/Results: Our unified model enables controllable generation of thumbnails, full-resolution images (4096×4096), and 4× super-resolution outputs within a single architecture. Experiments demonstrate state-of-the-art performance on multi-scale generation—especially in data-scarce full-image thumbnail synthesis—yielding globally consistent, detail-rich results. Moreover, the learned multi-scale features exhibit strong generalization in multiple-instance learning (MIL) tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Diffusion models have revolutionized image generation, yet several challenges restrict their application to large-image domains, such as digital pathology and satellite imagery. Given that it is infeasible to directly train a model on 'whole' images from domains with potential gigapixel sizes, diffusion-based generative methods have focused on synthesizing small, fixed-size patches extracted from these images. However, generating small patches has limited applicability since patch-based models fail to capture the global structures and wider context of large images, which can be crucial for synthesizing (semantically) accurate samples. In this paper, to overcome this limitation, we present ZoomLDM, a diffusion model tailored for generating images across multiple scales. Central to our approach is a novel magnification-aware conditioning mechanism that utilizes self-supervised learning (SSL) embeddings and allows the diffusion model to synthesize images at different 'zoom' levels, i.e., fixed-size patches extracted from large images at varying scales. ZoomLDM achieves state-of-the-art image generation quality across all scales, excelling particularly in the data-scarce setting of generating thumbnails of entire large images. The multi-scale nature of ZoomLDM unlocks additional capabilities in large image generation, enabling computationally tractable and globally coherent image synthesis up to $4096 imes 4096$ pixels and $4 imes$ super-resolution. Additionally, multi-scale features extracted from ZoomLDM are highly effective in multiple instance learning experiments. We provide high-resolution examples of the generated images on our website https://histodiffusion.github.io/docs/publications/zoomldm/.

Problem

• Generating large images with global coherence and context
• Overcoming patch-based limitations in multi-scale image synthesis
• Enabling high-resolution image generation with computational efficiency

Innovation

• Multi-scale image generation with diffusion model
• Magnification-aware conditioning using SSL embeddings
• Generates globally coherent large images efficiently