DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation

2024-12-10

arXiv.org

AI Summary

Existing text-to-image models struggle to simultaneously ensure character appearance consistency, controllable interactive layout, and dynamic expression/pose modulation in multi-character manga generation. To address this, we propose the novel task of *custom manga generation* and introduce a collaborative diffusion–multimodal large language model (MLLM) architecture. Our method features: (1) a novel masked cross-attention mechanism that explicitly fuses character identity and spatial relational features; (2) an MLLM-driven identity adapter enabling panel-level fine-grained semantic control; and (3) MangaZero—a large-scale manga dataset comprising 43K pages and 427K annotated panels. Experiments demonstrate significant improvements over state-of-the-art methods in multi-character consistency, layout controllability, and expressive diversity. Our framework supports end-to-end, text-driven generation of high-quality, dynamically coherent manga sequences.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Story visualization, the task of creating visual narratives from textual descriptions, has seen progress with text-to-image generation models. However, these models often lack effective control over character appearances and interactions, particularly in multi-character scenes. To address these limitations, we propose a new task: extbf{customized manga generation} and introduce extbf{DiffSensei}, an innovative framework specifically designed for generating manga with dynamic multi-character control. DiffSensei integrates a diffusion-based image generator with a multimodal large language model (MLLM) that acts as a text-compatible identity adapter. Our approach employs masked cross-attention to seamlessly incorporate character features, enabling precise layout control without direct pixel transfer. Additionally, the MLLM-based adapter adjusts character features to align with panel-specific text cues, allowing flexible adjustments in character expressions, poses, and actions. We also introduce extbf{MangaZero}, a large-scale dataset tailored to this task, containing 43,264 manga pages and 427,147 annotated panels, supporting the visualization of varied character interactions and movements across sequential frames. Extensive experiments demonstrate that DiffSensei outperforms existing models, marking a significant advancement in manga generation by enabling text-adaptable character customization. The project page is https://jianzongwu.github.io/projects/diffsensei/.

Problem

• Lack of control in character appearances and interactions in multi-character scenes.
• Need for precise layout control in manga generation without direct pixel transfer.
• Requirement for flexible adjustments in character expressions, poses, and actions.

Innovation

• Integrates diffusion models with multimodal LLMs
• Uses masked cross-attention for character feature control
• Introduces MangaZero dataset for manga generation