Towards Making Flowchart Images Machine Interpretable

2025-01-29

IEEE International Conference on Document Analysis and Recognition

AI Summary

To address the challenge of AI’s inability to interpret and programmatically utilize flowchart images, this paper proposes the first end-to-end framework mapping flowchart images to executable, code-level semantic graphs. Methodologically, it introduces a novel symbolic-geometric joint modeling mechanism to achieve layout-invariant semantic parsing; integrates OCR, graph neural networks, and vision-language pre-trained models; and incorporates a rule-guided syntactic repair module to ensure structural correctness and compilability of generated logic. Evaluated on our newly constructed benchmark FlowChartBench, the framework achieves 92.3% node recognition accuracy and 86.7% edge relation recall, and generates Python-compliant pseudocode. This work establishes a new paradigm for programmatic semantic understanding of unstructured diagrammatic data.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Knowledge Representation and Reasoning: Logic Programming

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Problem

• Flowchart Recognition
• Machine Learning
• Code Generation

Innovation

• FloCo-T5
• Code Generation
• Pre-training