UniWorld: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation

2025-06-03

AI Summary

Existing unified vision models excel at understanding and generation tasks but underperform on operational tasks such as image perception and editing. To address this gap, we propose the first high-resolution unified visual framework grounded in semantic features—departing from conventional VAE-based paradigms—and empirically demonstrate, for the first time, that a semantic encoder can replace VAEs to enable high-fidelity image manipulation. Our method integrates multimodal large language model (MLLM)-driven feature extraction, contrastive learning-based semantic encoding, and a lightweight diffusion decoder, optimized end-to-end in a shared semantic space. Trained on only 2.7M samples, our model achieves state-of-the-art performance across four core capabilities: visual understanding, image generation, pixel-level editing, and perceptual reasoning—surpassing leading unified models in all categories. We fully open-source the model weights, training code, and dataset.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Although existing unified models achieve strong performance in vision-language understanding and text-to-image generation, they remain limited in addressing image perception and manipulation -- capabilities increasingly demanded in practical applications. Recently, OpenAI introduced the powerful GPT-4o-Image model, which showcases advanced capabilities in comprehensive image perception and manipulation, sparking widespread interest. Through carefully designed experiments, we observe that GPT-4o-Image likely relies on semantic encoders rather than VAEs for feature extraction, despite VAEs being commonly regarded as crucial for image manipulation tasks. Inspired by this insight, we propose UniWorld, a unified generative framework built upon semantic features extracted from powerful multimodal large language models and contrastive semantic encoders. Using only 2.7M training data, UniWorld achieves impressive performance across diverse tasks, including image understanding, generation, manipulation, and perception. We fully open-source the UniWorld framework, including model weights, training and evaluation scripts, and datasets to promote reproducibility and further research.

Problem

• Unified model lacks image perception and manipulation tasks
• GPT-4o-Image uses semantic encoders instead of VAEs
• UniWorld improves image editing with minimal data usage

Innovation

• Uses semantic encoders for unified tasks
• Leverages visual-language model features
• Achieves strong performance with minimal data