Linking heterogeneous microstructure informatics with expert characterization knowledge through customized and hybrid vision-language representations for industrial qualification

2025-08-27

AI Summary

Addressing the challenge of rapid and reliable certification for additively manufactured heterogeneous microstructural materials, this paper proposes a zero-shot vision-language representation framework that jointly leverages microscopic images and expert textual knowledge. Methodologically, it constructs a cross-modal shared embedding space by integrating deep semantic segmentation, pre-trained multimodal models (CLIP/FLAVA), and Z-score normalization, and introduces a similarity-based hybrid representation strategy enabling reference-based positive/negative sample matching and human-in-the-loop decision-making without fine-tuning. Its key innovation lies in the first zero-shot incorporation of domain-expert knowledge into microstructural certification—significantly enhancing traceability and interpretability. Evaluated on a metal composite dataset, the framework achieves accurate discrimination between conforming and defective samples: FLAVA demonstrates superior visual discriminability, whereas CLIP excels in text–semantic alignment.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Rapid and reliable qualification of advanced materials remains a bottleneck in industrial manufacturing, particularly for heterogeneous structures produced via non-conventional additive manufacturing processes. This study introduces a novel framework that links microstructure informatics with a range of expert characterization knowledge using customized and hybrid vision-language representations (VLRs). By integrating deep semantic segmentation with pre-trained multi-modal models (CLIP and FLAVA), we encode both visual microstructural data and textual expert assessments into shared representations. To overcome limitations in general-purpose embeddings, we develop a customized similarity-based representation that incorporates both positive and negative references from expert-annotated images and their associated textual descriptions. This allows zero-shot classification of previously unseen microstructures through a net similarity scoring approach. Validation on an additively manufactured metal matrix composite dataset demonstrates the framework's ability to distinguish between acceptable and defective samples across a range of characterization criteria. Comparative analysis reveals that FLAVA model offers higher visual sensitivity, while the CLIP model provides consistent alignment with the textual criteria. Z-score normalization adjusts raw unimodal and cross-modal similarity scores based on their local dataset-driven distributions, enabling more effective alignment and classification in the hybrid vision-language framework. The proposed method enhances traceability and interpretability in qualification pipelines by enabling human-in-the-loop decision-making without task-specific model retraining. By advancing semantic interoperability between raw data and expert knowledge, this work contributes toward scalable and domain-adaptable qualification strategies in engineering informatics.

Problem

• Linking microstructure informatics with expert characterization knowledge
• Enabling zero-shot classification of unseen microstructures
• Distinguishing acceptable and defective samples through hybrid representations

Innovation

• Customized hybrid vision-language representations for microstructures
• Zero-shot classification using similarity-based scoring approach
• Integration of semantic segmentation with multimodal models