SDIGLM: Leveraging Large Language Models and Multi-Modal Chain of Thought for Structural Damage Identification

2025-04-12

AI Summary

Existing vision models exhibit weak fine-grained damage-type discrimination and lack natural language generation capabilities for structural damage identification in civil engineering. Method: This paper introduces the first multimodal large language model (MLLM) tailored for civil infrastructure damage analysis. It proposes a dual-path collaborative framework: (i) “U-Net segmentation + visual chain-of-reasoning” for precise localization and logical visual inference, and (ii) “prompt-engineering-driven linguistic chain-of-reasoning” for structured textual description. Built upon VisualGLM-6B, the model supports cross-modal alignment and interpretable damage reasoning via multi-turn dialogue fine-tuning and multimodal chain-of-thought inference. Contribution/Results: The model achieves 95.24% classification accuracy while generating fine-grained natural language descriptions—including cavity dimensions, crack orientation, and corrosion severity—thereby bridging a critical gap in structural health monitoring and significantly enhancing human-AI collaborative decision-making.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Existing computer vision(CV)-based structural damage identification models demonstrate notable accuracy in categorizing and localizing damage. However, these models present several critical limitations that hinder their practical application in civil engineering(CE). Primarily, their ability to recognize damage types remains constrained, preventing comprehensive analysis of the highly varied and complex conditions encountered in real-world CE structures. Second, these models lack linguistic capabilities, rendering them unable to articulate structural damage characteristics through natural language descriptions. With the continuous advancement of artificial intelligence(AI), large multi-modal models(LMMs) have emerged as a transformative solution, enabling the unified encoding and alignment of textual and visual data. These models can autonomously generate detailed descriptive narratives of structural damage while demonstrating robust generalization across diverse scenarios and tasks. This study introduces SDIGLM, an innovative LMM for structural damage identification, developed based on the open-source VisualGLM-6B architecture. To address the challenge of adapting LMMs to the intricate and varied operating conditions in CE, this work integrates a U-Net-based semantic segmentation module to generate defect segmentation maps as visual Chain of Thought(CoT). Additionally, a multi-round dialogue fine-tuning dataset is constructed to enhance logical reasoning, complemented by a language CoT formed through prompt engineering. By leveraging this multi-modal CoT, SDIGLM surpasses general-purpose LMMs in structural damage identification, achieving an accuracy of 95.24% across various infrastructure types. Moreover, the model effectively describes damage characteristics such as hole size, crack direction, and corrosion severity.

Problem

• Limited damage type recognition in CV-based structural identification models
• Lack of linguistic capabilities for describing structural damage characteristics
• Adapting LMMs to complex civil engineering operating conditions

Innovation

• Uses VisualGLM-6B for structural damage identification
• Integrates U-Net for defect segmentation maps
• Employs multi-modal Chain of Thought for accuracy