Structured Extraction from Business Process Diagrams Using Vision-Language Models

2025-11-27

AI Summary

When BPMN process diagram source files (e.g., XML) are unavailable, recovering structured semantic information directly from raster images remains challenging. Method: We propose an end-to-end vision-language joint approach that tightly integrates multimodal large models (VLMs) with optical character recognition (OCR) via prompt engineering—enabling unified modeling of graphical symbol recognition, text localization, and semantic alignment without manual annotations or textual priors. Contribution/Results: Ablation studies and statistical analysis across multiple VLM benchmarks demonstrate that OCR enhancement significantly improves node-type identification and control-flow relation extraction accuracy (average +12.7%). The method exhibits strong robustness against image degradation—including blurriness, scaling artifacts, and low resolution. This work establishes a practical, deployable paradigm for structured image parsing in reverse engineering and legacy system digitization.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Business Process Model and Notation (BPMN) is a widely adopted standard for representing complex business workflows. While BPMN diagrams are often exchanged as visual images, existing methods primarily rely on XML representations for computational analysis. In this work, we present a pipeline that leverages Vision-Language Models (VLMs) to extract structured JSON representations of BPMN diagrams directly from images, without requiring source model files or textual annotations. We also incorporate optical character recognition (OCR) for textual enrichment and evaluate the generated element lists against ground truth data derived from the source XML files. Our approach enables robust component extraction in scenarios where original source files are unavailable. We benchmark multiple VLMs and observe performance improvements in several models when OCR is used for text enrichment. In addition, we conducted extensive statistical analyses of OCR-based enrichment methods and prompt ablation studies, providing a clearer understanding of their impact on model performance.

Problem

• Extract structured JSON from BPMN diagram images
• Leverage Vision-Language Models without source files
• Enrich extraction using OCR and evaluate accuracy

Innovation

• Vision-Language Models extract structured JSON from BPMN images
• OCR enriches text to improve model performance
• Pipeline works without source files or annotations