Predicting Performance of Object Detection Models in Electron Microscopy Using Random Forests

2025-01-14

AI Summary

Evaluating the generalization capability of small-defect detection models on transmission electron microscopy (TEM) images is challenging due to reliance on scarce, labor-intensive ground-truth annotations. Method: This paper proposes a label-free, unsupervised method to estimate the F1 score of object detectors (e.g., YOLO, Faster R-CNN) by leveraging prediction-level features—particularly confidence scores—and training a random forest regressor to directly predict F1 on unseen TEM images. The approach enables domain-shift diagnosis and model suitability assessment across varying imaging conditions and material systems. Contribution/Results: Validated on three diverse TEM datasets, the method achieves a mean absolute error (MAE) of 0.09 and an R² of 0.77 on test sets, demonstrating robustness and generalizability. Its core innovation lies in repurposing detector confidence features for regression-based performance estimation—enabling rapid, annotation-free, and domain-agnostic assessment of detection reliability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Object Detection & Categorization
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Quantifying prediction uncertainty when applying object detection models to new, unlabeled datasets is critical in applied machine learning. This study introduces an approach to estimate the performance of deep learning-based object detection models for quantifying defects in transmission electron microscopy (TEM) images, focusing on detecting irradiation-induced cavities in TEM images of metal alloys. We developed a random forest regression model that predicts the object detection F1 score, a statistical metric used to evaluate the ability to accurately locate and classify objects of interest. The random forest model uses features extracted from the predictions of the object detection model whose uncertainty is being quantified, enabling fast prediction on new, unlabeled images. The mean absolute error (MAE) for predicting F1 of the trained model on test data is 0.09, and the $R^2$ score is 0.77, indicating there is a significant correlation between the random forest regression model predicted and true defect detection F1 scores. The approach is shown to be robust across three distinct TEM image datasets with varying imaging and material domains. Our approach enables users to estimate the reliability of a defect detection and segmentation model predictions and assess the applicability of the model to their specific datasets, providing valuable information about possible domain shifts and whether the model needs to be fine-tuned or trained on additional data to be maximally effective for the desired use case.

Problem

• Transmission Electron Microscopy
• Small Defect Detection
• Target Detection Model

Innovation

• Random Forest Model
• F1 Score Prediction
• Transmission Electron Microscopy (TEM)