Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement

2026-03-23

AI Summary

Existing text-to-layout generation methods rely on a code-only paradigm, lacking awareness of the visual appearance after rendering and struggling to balance readability with aesthetic quality. This work proposes the Visual Feedback Layout Model (VFLM), which introduces a visual feedback mechanism into layout generation for the first time, establishing a multimodal framework capable of self-reflection and iterative refinement. VFLM integrates a multimodal large language model, reinforcement learning, and a vision-based reward model grounded in OCR accuracy, using only the readability of the final rendered image as the reward signal to drive optimization. Experiments demonstrate that VFLM significantly outperforms state-of-the-art multimodal large language models, specialized layout generators, and code-only baselines across multiple benchmarks, underscoring the critical role of visual feedback in design-oriented generative tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have enabled automated generation of structured layouts from natural language descriptions. Existing methods typically follow a code-only paradigm that generates code to represent layouts, which are then rendered by graphic engines to produce final images. However, they are blind to the rendered visual outcome, making it difficult to guarantee readability and aesthetics. In this paper, we identify visual feedback as a critical factor in layout generation and propose Visual Feedback Layout Model (VFLM), a self-improving framework that leverages visual feedback iterative refinement. VFLM is capable of performing adaptive reflective generation, which leverages visual information to reflect on previous issues and iteratively generates outputs until satisfactory quality is achieved. It is achieved through reinforcement learning with a visually grounded reward model that incorporates OCR accuracy. By rewarding only the final generated outcome, we can effectively stimulate the model's iterative and reflective generative capabilities. Experiments across multiple benchmarks show that VFLM consistently outperforms advanced MLLMs, existing layout models, and code-only baselines, establishing visual feedback as critical for design-oriented MLLMs. Our code and data are available at https://github.com/FolSpark/VFLM.

Problem

• visual feedback
• text layout
• readability
• aesthetics
• multimodal large language models

Innovation

• visual feedback
• iterative refinement
• multimodal large language models
• layout generation
• reinforcement learning