From "What" to "How": Constrained Reasoning for Autoregressive Image Generation

2026-03-03

AI Summary

Existing autoregressive image generation methods primarily focus on “what to draw” while neglecting the reasoning of “how to compose,” often resulting in ambiguous spatial relationships and implausible object overlaps. This work proposes CoR-Painter, a novel framework that introduces a “how-to-draw → what-to-draw” generation paradigm: it first infers visual constraints—such as spatial relations and compositional rules—from textual prompts, then produces structurally coherent, detailed descriptions to guide image synthesis. The approach integrates constraint reasoning into the autoregressive generation process and employs a dual-objective GRPO strategy to jointly optimize textual reasoning and visual mapping. Evaluated on T2I-CompBench, GenEval, and WISE benchmarks, CoR-Painter achieves state-of-the-art performance, with a 5.41% improvement on spatial metrics in T2I-CompBench.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Natural Language Processing: Generation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Autoregressive image generation has seen recent improvements with the introduction of chain-of-thought and reinforcement learning. However, current methods merely specify "What" details to depict by rewriting the input prompt, yet fundamentally fail to reason about "How" to structure the overall image. This inherent limitation gives rise to persistent issues, such as spatial ambiguity directly causing unrealistic object overlaps. To bridge this gap, we propose CoR-Painter, a novel framework that pioneers a "How-to-What" paradigm by introducing Constrained Reasoning to guide the autoregressive generation. Specifically, it first deduces "How to draw" by deriving a set of visual constraints from the input prompt, which explicitly govern spatial relationships, key attributes, and compositional rules. These constraints steer the subsequent generation of a detailed description "What to draw", providing a structurally sound and coherent basis for accurate visual synthesis. Additionally, we introduce a Dual-Objective GRPO strategy that specifically optimizes the textual constrained reasoning and visual projection processes to ensure the coherence and quality of the entire generation pipeline. Extensive experiments on T2I-CompBench, GenEval, and WISE demonstrate that our method achieves state-of-the-art performance, with significant improvements in spatial metrics (e.g., +5.41% on T2I-CompBench).

Problem

• autoregressive image generation
• spatial ambiguity
• constrained reasoning
• image composition
• visual synthesis

Innovation

• Constrained Reasoning
• Autoregressive Image Generation
• Visual Constraints
• How-to-What Paradigm
• Dual-Objective GRPO