QuPAINT: Physics-Aware Instruction Tuning Approach to Quantum Material Discovery

2026-02-19

AI Summary

This work addresses the challenges of identifying two-dimensional quantum materials from optical microscopy images, which include low interlayer contrast, scarcity of annotated data, and poor generalization across diverse experimental conditions. To overcome these limitations, the authors propose a physics-aware multimodal framework that integrates a thin-film interference–based synthetic data generator (Synthia), a physics-informed attention mechanism, and instruction tuning of multimodal large language models. They also introduce QMat-Instruct, the first large-scale instruction dataset for quantum materials, along with QF-Bench, a comprehensive evaluation benchmark. The proposed approach substantially reduces reliance on manual annotations and achieves more robust and generalizable recognition performance across various materials, substrates, and imaging conditions, thereby advancing the integration of physical knowledge with multimodal foundation models.

Technology Category

• Machine Learning: Quantum Machine Learning
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Characterizing two-dimensional quantum materials from optical microscopy images is challenging due to the subtle layer-dependent contrast, limited labeled data, and significant variation across laboratories and imaging setups. Existing vision models struggle in this domain since they lack physical priors and cannot generalize to new materials or hardware conditions. This work presents a new physics-aware multimodal framework that addresses these limitations from both the data and model perspectives. We first present Synthia, a physics-based synthetic data generator that simulates realistic optical responses of quantum material flakes under thin-film interference. Synthia produces diverse and high-quality samples, helping reduce the dependence on expert manual annotation. We introduce QMat-Instruct, the first large-scale instruction dataset for quantum materials, comprising multimodal, physics-informed question-answer pairs designed to teach Multimodal Large Language Models (MLLMs) to understand the appearance and thickness of flakes. Then, we propose Physics-Aware Instruction Tuning (QuPAINT), a multimodal architecture that incorporates a Physics-Informed Attention module to fuse visual embeddings with optical priors, enabling more robust and discriminative flake representations. Finally, we establish QF-Bench, a comprehensive benchmark spanning multiple materials, substrates, and imaging settings, offering standardized protocols for fair and reproducible evaluation.

Problem

• quantum materials
• optical microscopy
• layer-dependent contrast
• limited labeled data
• generalization across imaging setups

Innovation

• Physics-Aware Instruction Tuning
• Synthetic Data Generation
• Multimodal Large Language Models
• Thin-Film Interference
• Quantum Material Discovery