Jodi: Unification of Visual Generation and Understanding via Joint Modeling

2025-05-25

AI Summary

This work addresses the longstanding disjunction between generation and understanding in traditional vision tasks by proposing Jodi, the first unified diffusion framework for joint visual generation and understanding. Methodologically, Jodi introduces a role-switching mechanism and a linear-diffusion Transformer architecture to enable bidirectional co-modeling; constructs Joint-1.6M—a novel benchmark comprising 7 multi-label categories; and integrates LLM-enhanced annotation with cross-domain conditional sampling to support image–multi-label joint generation, label-controllable synthesis, and multi-domain semantic awareness. Experiments demonstrate that Jodi consistently outperforms task-specific baselines in both generation quality (lower FID) and understanding accuracy (higher mAP), enables arbitrary multi-label composition control, and performs one-shot multi-domain label prediction with strong generalization—establishing a new paradigm for vision foundation models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Visual generation and understanding are two deeply interconnected aspects of human intelligence, yet they have been traditionally treated as separate tasks in machine learning. In this paper, we propose Jodi, a diffusion framework that unifies visual generation and understanding by jointly modeling the image domain and multiple label domains. Specifically, Jodi is built upon a linear diffusion transformer along with a role switch mechanism, which enables it to perform three particular types of tasks: (1) joint generation, where the model simultaneously generates images and multiple labels; (2) controllable generation, where images are generated conditioned on any combination of labels; and (3) image perception, where multiple labels can be predicted at once from a given image. Furthermore, we present the Joint-1.6M dataset, which contains 200,000 high-quality images collected from public sources, automatic labels for 7 visual domains, and LLM-generated captions. Extensive experiments demonstrate that Jodi excels in both generation and understanding tasks and exhibits strong extensibility to a wider range of visual domains. Code is available at https://github.com/VIPL-GENUN/Jodi.

Problem

• Unifying visual generation and understanding via joint modeling
• Enabling joint, controllable generation and multi-label image perception
• Addressing the separation of visual tasks in machine learning

Innovation

• Unifies visual generation and understanding via joint modeling
• Uses linear diffusion transformer with role switch
• Introduces Joint-1.6M dataset for multi-domain tasks