UniSpace: Unified Visual Representation and Scalable Multimodal Modeling

2026-08-09

AI Summary

Existing semantic visual encoders often lose fine-grained details in their final tokens, hindering high-fidelity image generation and editing. This work proposes UniSpace, a unified vision model that, built upon a frozen pretrained Vision Transformer (ViT), introduces a Patch Reparameterization mechanism to inject reconstruction-aware patch embeddings while preserving semantic understanding capabilities. UniSpace further integrates a Mixture-of-Transformer-Experts architecture, enabling, for the first time, the unification of multimodal understanding, image generation, and instruction-guided editing within a single visual representation space—without requiring a separate VAE pathway. Evaluated at system level with 8 billion parameters, UniSpace demonstrates high-quality text-to-image synthesis and editing performance, validating the feasibility and superiority of a unified visual interface.

Abstract

Semantic vision encoders have become a central visual interface for multimodal understanding and semantic conditioning in image generation. However, their final tokens discard fine-grained visual details, leading to poor pixel reconstruction and limiting their use in reconstruction-sensitive tasks such as image generation and editing. In this work, we ask whether understanding, generation, and editing can be modeled in a single visual representation space built from a pretrained semantic ViT. We show that the frozen Transformer blocks of a semantic ViT are not intrinsically unable to preserve visual details. Instead, the original patch parameterization drives the representation toward semantic abstraction, making fine-grained information difficult to recover from the final tokens. Based on this observation, we introduce \emph{Patch Reparameterization}, which preserves the original semantic pathway while adding a reconstruction-aware patch embedding that provides fine-grained visual information to the same frozen ViT blocks. The resulting unified representation preserves multimodal understanding while enabling high-fidelity image reconstruction and a favorable reconstruction--generation trade-off. We further scale this representation into \emph{UniSpace}, an 8B Mixture-of-Transformer-Experts model that performs understanding, generation, and editing in the same visual space without a separate VAE pathway. System-level evaluations demonstrate practical text-to-image generation and instruction-based image editing, showing that a reparameterized pretrained ViT can serve as a unified visual interface for scalable multimodal modeling.

Problem

• semantic vision encoders
• fine-grained visual details
• image reconstruction
• multimodal modeling
• visual representation

Innovation

• Patch Reparameterization
• Unified Visual Representation
• Semantic ViT
• Mixture-of-Experts
• Multimodal Modeling