Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generation

2026-05-12

AI Summary

Existing approaches struggle to model long-range dependencies due to the structural separation between images and text, leading to degraded generation quality under complex interleaved instructions. This work proposes INSET, a novel model that, for the first time, embeds images as native, dense language tokens directly into the text sequence, enabling unified, end-to-end multimodal generation and inherently supporting multimodal editing. Built upon a Transformer architecture, INSET integrates vision-language modeling with large language models to construct a scalable data engine that synthesizes 15 million high-quality interleaved samples. Evaluated on InterleaveBench, INSET significantly outperforms current methods, particularly in multi-image consistency and text alignment, with its advantage becoming more pronounced as instruction complexity increases.

Abstract

While recent advancements in multimodal language models have enabled image generation from expressive multi-image instructions, existing methods struggle to maintain performance under complex interleaved instructions. This limitation stems from the structural separation of images and text in current paradigms, which forces models to bridge difficult long-range dependencies to match descriptions with visual targets. To address these challenges, we propose \texttt{I}mages i\texttt{N} \texttt{SE}n\texttt{T}ences (\textit{a.k.a}, INSET), a unified generation model that seamlessly embeds images as native vocabulary within textual instructions. By positioning visual features directly at their corresponding semantic slots, INSET leverages the contextual locality of transformers for precise object binding, effectively treating images as dense, expressive language tokens. Furthermore, we introduce a scalable data engine that synthesizes 15M high-quality interleaved samples from standard image and video datasets, utilizing VLMs and LLMs to construct rich, long-horizon sequences. Evaluation results on InterleaveBench demonstrate that INSET significantly outperforms state-of-the-art methods in multi-image consistency and text alignment, with performance gaps widening as input complexity increases. Beyond standard generation, our approach inherently extends to multimodal image editing, integrating visual content as part of the instruction to facilitate highly expressive and creative visual manipulations.

Problem

• interleaved instructions
• multimodal generation
• image-text alignment
• visual generation
• long-range dependencies

Innovation

• interleaved multimodal instructions
• unified visual generation
• image-as-token
• contextual locality
• multimodal image editing