JetFormer: An Autoregressive Generative Model of Raw Images and Text

2024-11-29

arXiv.org

AI Summary

This work addresses the limitation of existing image-text joint generation models that rely on pre-trained encoders/decoders. We propose an end-to-end autoregressive framework that jointly models raw pixels and text tokens without discrete representation modules such as VQ-VAE or VAE. Our key contributions are: (1) a learnable invertible normalizing flow to construct continuous soft token representations, enabling high-fidelity image generation while improving likelihood estimation accuracy; and (2) a decoder-only Transformer architecture for multimodal joint autoregressive training. Experiments demonstrate that our method matches state-of-the-art VQ-VAE/VAE-based baselines in image generation quality, significantly enhances robustness in image understanding tasks, and achieves the current best log-likelihood lower bound on image data.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Removing modeling constraints and unifying architectures across domains has been a key driver of the recent progress in training large multimodal models. However, most of these models still rely on many separately trained components such as modality-specific encoders and decoders. In this work, we further streamline joint generative modeling of images and text. We propose an autoregressive decoder-only transformer - JetFormer - which is trained to directly maximize the likelihood of raw data, without relying on any separately pretrained components, and can understand and generate both text and images. Specifically, we leverage a normalizing flow model to obtain a soft-token image representation that is jointly trained with an autoregressive multimodal transformer. The normalizing flow model serves as both an image encoder for perception tasks and an image decoder for image generation tasks during inference. JetFormer achieves text-to-image generation quality competitive with recent VQ-VAE- and VAE-based baselines. These baselines rely on pretrained image autoencoders, which are trained with a complex mixture of losses, including perceptual ones. At the same time, JetFormer demonstrates robust image understanding capabilities. To the best of our knowledge, JetFormer is the first model that is capable of generating high-fidelity images and producing strong log-likelihood bounds.

Problem

• Unify image and text modeling without separate components
• Generate high-fidelity images via autoregressive likelihood maximization
• Achieve multimodal understanding without pretrained encoders/decoders

Innovation

• Autoregressive decoder-only transformer for multimodal data
• Normalizing flow model for soft-token image representation
• Joint training without pretrained components