FabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queries

2024-07-15

arXiv.org

AI Summary

To address the challenges of detecting minute defects in wafer scanning electron microscopy (SEM) images, the subjectivity of manual thresholding, and insufficient incorporation of domain knowledge in integrated circuit manufacturing, this paper proposes a large vision-language model framework tailored for wafer defect analysis. Methodologically, we design a lightweight modulation module and an interactive corpus training strategy, integrating domain-knowledge injection, cross-modal alignment, and process-aware fine-tuning to mitigate multimodal bias and enhance joint defect–semantic modeling. Our key contributions are: (1) the first end-to-end unified modeling framework supporting defect detection, root-cause reasoning, and process-expert question answering; and (2) on real-world production-line data, achieving a 12.3% improvement in defect detection mAP and an 18.7% gain in knowledge-based QA accuracy—significantly improving diagnostic interpretability and automation capability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Intelligence is key to advancing integrated circuit (IC) fabrication. Recent breakthroughs in Large Multimodal Models (LMMs) have unlocked extraditionary abilities in understanding images and text, fostering intelligent fabrication. Leveraging the power of LMMs, we introduce FabGPT, a customized IC fabrication large multimodal model for wafer defect knowledge query. FabGPT manifests expertise in conducting defect detection in Scanning Electron Microscope (SEM) images, performing root cause analysis, and providing expert Q&A on fabrication processes. FabGPT matches enhanced multimodal features to automatically detect minute defects under complex wafer backgrounds and reduce the subjectivity of manual threshold settings. Besides, the proposed modulation module and interactive corpus training strategy embed wafer defect knowledge into the pre-trained model, effectively balancing Q&A queries related to defect knowledge and original knowledge and mitigating the modality bias issues. Experiments on in-house fab data show that FabGPT achieves significant performance improvement in wafer defect detection and knowledge querying.

Problem

• Enhances wafer defect detection efficiency
• Reduces manual threshold setting subjectivity
• Balances defect and original knowledge queries

Innovation

• Customized IC fabrication multimodal model
• Enhanced defect detection in SEM images
• Modulation module for knowledge integration