Enhance Multimodal Consistency and Coherence for Text-Image Plan Generation

2025-06-13

AI Summary

Prior work on large language models (LLMs) primarily focuses on text-only task planning, overlooking their potential for text-image multimodal plan generation—particularly hindered by cross-modal misalignment and weak visual step coherence. Method: We propose an iterative text-image co-generation and refinement framework that supports plug-and-play integration of leading LLMs (e.g., Mistral-7B, Gemini-1.5, GPT-4o), incorporating LLM-based reasoning, PDDL-style visual semantic parsing, visual editing, and multi-round cross-modal feedback refinement. Contribution/Results: We introduce MPlanner—the first thousand-scale text-image planning benchmark—and a dedicated evaluation protocol. Our approach achieves significant improvements in multimodal consistency and step coherence across 11 everyday task categories, outperforming strong baselines. The code and dataset are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

People get informed of a daily task plan through diverse media involving both texts and images. However, most prior research only focuses on LLM's capability of textual plan generation. The potential of large-scale models in providing text-image plans remains understudied. Generating high-quality text-image plans faces two main challenges: ensuring consistent alignment between two modalities and keeping coherence among visual steps. To address these challenges, we propose a novel framework that generates and refines text-image plans step-by-step. At each iteration, our framework (1) drafts the next textual step based on the prediction history; (2) edits the last visual step to obtain the next one; (3) extracts PDDL-like visual information; and (4) refines the draft with the extracted visual information. The textual and visual step produced in stage (4) and (2) will then serve as inputs for the next iteration. Our approach offers a plug-and-play improvement to various backbone models, such as Mistral-7B, Gemini-1.5, and GPT-4o. To evaluate the effectiveness of our approach, we collect a new benchmark consisting of 1,100 tasks and their text-image pair solutions covering 11 daily topics. We also design and validate a new set of metrics to evaluate the multimodal consistency and coherence in text-image plans. Extensive experiment results show the effectiveness of our approach on a range of backbone models against competitive baselines. Our code and data are available at https://github.com/psunlpgroup/MPlanner.

Problem

• Ensures consistent alignment between text and image modalities
• Maintains coherence among visual steps in plans
• Generates high-quality multimodal text-image task plans

Innovation

• Step-by-step text-image plan generation framework
• PDDL-like visual information extraction for refinement
• Plug-and-play compatibility with multiple backbone models