Flowchart2Mermaid: A Vision-Language Model Powered System for Converting Flowcharts into Editable Diagram Code

2025-12-01

AI Summary

This work addresses the challenge of editing and reusing static flowchart images. We propose a lightweight web-based system that enables end-to-end conversion from flowchart images to version-controllable Mermaid.js code. Methodologically, we introduce the first vision-language model (VLM)–prompt engineering co-generation framework tailored for flowchart images, integrating Mermaid.js rendering with an AI-driven natural language understanding module. The system supports inline editing, drag-and-drop insertion, and instruction-based interaction, enabling real-time bidirectional synchronization between diagrams and code. Key contributions include: (1) the first VLM–prompt collaborative generation framework for flowchart image parsing; (2) a multi-dimensional evaluation metric covering structural accuracy, procedural logic, syntactic validity, and completeness; and (3) empirical validation of generalization across multiple state-of-the-art VLMs. Experiments demonstrate high syntactic correctness of generated Mermaid.js code, strict diagram–code consistency, and significant improvements in editing efficiency and reusability.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Flowcharts are common tools for communicating processes but are often shared as static images that cannot be easily edited or reused. We present extsc{Flowchart2Mermaid}, a lightweight web system that converts flowchart images into editable Mermaid.js code which is a markup language for visual workflows, using a detailed system prompt and vision-language models. The interface supports mixed-initiative refinement through inline text editing, drag-and-drop node insertion, and natural-language commands interpreted by an integrated AI assistant. Unlike prior image-to-diagram tools, our approach produces a structured, version-controllable textual representation that remains synchronized with the rendered diagram. We further introduce evaluation metrics to assess structural accuracy, flow correctness, syntax validity, and completeness across multiple models.

Problem

• Converts static flowchart images into editable Mermaid.js code
• Enables mixed-initiative refinement via text editing and AI commands
• Produces structured, version-controllable textual diagram representations

Innovation

• Vision-language model converts flowchart images to editable code
• Mixed-initiative interface enables text, drag-drop, and natural-language refinement
• System produces structured, version-controllable textual representation synchronized with diagram