InstanceControl: Controllable Complex Image Generation without Instance Labeling

2026-06-30

AI Summary

This work addresses the limitations of existing controllable image generation methods, which often conflate instance attributes in complex multi-instance scenes and rely heavily on manual annotations. To overcome these challenges, we propose InstanceControl—the first approach to enable precise multi-instance control without requiring instance-level labels. Our method leverages a vision-language model (VLM) to automatically establish instance-level correspondences between textual descriptions and visual conditions, and introduces an adaptive mask refinement mechanism that dynamically optimizes instance masks during generation. Extensive experiments demonstrate that InstanceControl significantly outperforms current state-of-the-art techniques, achieving superior instance-level control accuracy while maintaining high-fidelity image synthesis.

Abstract

Controllable image generation methods, such as ControlNet, have demonstrated a remarkable capacity to introduce visual conditions(e.g., depth maps) to guide image generation. However, these methods often struggle with complex multi-instance scenes, frequently leading to attribute confusion among instances. While recent approaches attempt to mitigate this via manual instance labeling, such requirements are labor-intensive. In this paper, we propose InstanceControl, a novel multi-instance controllable generation method that eliminates the need for instance labeling. We identify the primary bottleneck in existing methods as the inability to accurately associate instance descriptions with their corresponding regions within visual conditions. To address this, we leverage the Vision-Language Model (VLM) to establish instance-level correspondences between text prompts and visual conditions. Specifically, the VLM automatically parses instance descriptions from the text prompts and simultaneously predicts instance masks based on the visual conditions. Furthermore, since the predicted masks may contain noise, we introduce an adaptive mask refinement strategy that dynamically refines these instance masks during the generation process. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods, achieving superior fidelity and precise instance-level control.

Problem

• controllable image generation
• multi-instance scenes
• instance labeling
• attribute confusion
• visual conditions

Innovation

• Instance-level control
• Vision-Language Model
• Controllable image generation
• Mask refinement
• Multi-instance generation