BiGR: Harnessing Binary Latent Codes for Image Generation and Improved Visual Representation Capabilities

2024-10-18

arXiv.org

AI Summary

This work addresses the limitations of existing image generation models in generation quality, visual representation capability, and multi-task generalization. We propose BiGR—the first conditional image generation model that unifies generative and discriminative capabilities within a single architecture. Its key contributions are: (1) a novel compact binary latent code modeling scheme, incorporating a binary tokenizer and masked autoregressive prediction; (2) joint optimization of diverse generative tasks—including inpainting, editing, outpainting, and interpolation—alongside discriminative evaluation via linear probing; and (3) entropy-ordered sampling for efficient, high-fidelity generation. Experiments demonstrate significant improvements: FID-50k substantially decreases, and linear probe accuracy increases markedly. BiGR enables zero-shot cross-task generalization and successfully extends to text-to-image generation, achieving state-of-the-art generation performance while preserving strong representation learning capacity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce BiGR, a novel conditional image generation model using compact binary latent codes for generative training, focusing on enhancing both generation and representation capabilities. BiGR is the first conditional generative model that unifies generation and discrimination within the same framework. BiGR features a binary tokenizer, a masked modeling mechanism, and a binary transcoder for binary code prediction. Additionally, we introduce a novel entropy-ordered sampling method to enable efficient image generation. Extensive experiments validate BiGR's superior performance in generation quality, as measured by FID-50k, and representation capabilities, as evidenced by linear-probe accuracy. Moreover, BiGR showcases zero-shot generalization across various vision tasks, enabling applications such as image inpainting, outpainting, editing, interpolation, and enrichment, without the need for structural modifications. Our findings suggest that BiGR unifies generative and discriminative tasks effectively, paving the way for further advancements in the field. We further enable BiGR to perform text-to-image generation, showcasing its potential for broader applications.

Problem

• Image Generation
• Image Understanding
• Text-to-Image Synthesis

Innovation

• Binary Encoding
• Transformer for Binary Code Prediction
• Integrative Image Generation and Assessment