Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models

2025-03-15

AI Summary

To address the challenge of automating the conversion of UML diagram images from legacy systems into executable code, this paper proposes an end-to-end solution based on domain-adapted multimodal large language models (MM-LLMs), supporting generation of standard UML code from activity and sequence diagram images. Our method introduces the first MM-LLM tailored for UML diagram understanding and constructs the first synthetic UML diagram dataset. We further demonstrate the effectiveness of LoRA-based fine-tuning under few-shot settings. Experimental results show substantial improvements over baselines: BLEU score of 0.779 and SSIM of 0.942 on sequence diagrams. This work establishes a scalable, low-intervention paradigm for UML reverse engineering—reducing modeling effort and accelerating legacy system modernization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The Unified Modeling Language is a standardized visual language widely used for modeling and documenting the design of software systems. Although many tools generate UML diagrams from UML code, generating executable UML code from image-based UML diagrams remains challenging. This paper proposes a new approach to generate UML code using a large multimodal language model automatically. Synthetic UML activity and sequence diagram datasets were created to train and test the model. We compared standard fine-tuning with LoRA techniques to optimize base models. The experiments measured code generation accuracy across different model sizes and training strategies. These results demonstrated that domain-adapted MM-LLMs perform for UML code generation automation, whereby, at the best model, it achieved BLEU and SSIM scores of 0.779 and 0.942 on sequence diagrams. This will enable the modernization of legacy systems and decrease the manual effort in software development workflows.

Problem

• Automates UML code generation from diagram images using multimodal models.
• Addresses challenges in converting image-based UML diagrams to executable code.
• Improves software development workflows by reducing manual effort and modernizing legacy systems.

Innovation

• UML code generation from diagram images
• Multimodal large language models for automation
• Domain-adapted MM-LLMs optimize UML code accuracy