Improving Object Detection by Modifying Synthetic Data with Explainable AI

2024-12-02

arXiv.org

AI Summary

To address the limited generalizability of infrared vehicle detection models caused by scarcity of real-world annotated data, this paper proposes an XAI-driven human-in-the-loop synthetic data optimization framework. The method leverages Grad-CAM saliency maps from YOLOv8 to identify model decision weaknesses, guiding targeted realism enhancement of critical vehicle regions and abstraction of distractive regions in Unity-based 3D mesh rendering. It introduces a novel bidirectional realism control mechanism that enables synthetic data to adaptively evolve according to model deficiencies. Integrating XAI interpretability, photorealistic 3D rendering, and iterative human-AI co-optimization, the framework significantly improves detection robustness. Evaluated on the ATR DSIAC infrared dataset, the baseline achieves 94.6% mAP₅₀ for unseen pose detection; after XAI-guided optimization, performance rises to 96.1% (+1.5%), with markedly reduced false positives and substantially lower manual hyperparameter tuning effort.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Limited real-world data severely impacts model performance in many computer vision domains, particularly for samples that are underrepresented in training. Synthetically generated images are a promising solution, but 1) it remains unclear how to design synthetic training data to optimally improve model performance (e.g, whether and where to introduce more realism or more abstraction) and 2) the domain expertise, time and effort required from human operators for this design and optimisation process represents a major practical challenge. Here we propose a novel conceptual approach to improve the efficiency of designing synthetic images, by using robust Explainable AI (XAI) techniques to guide a human-in-the-loop process of modifying 3D mesh models used to generate these images. Importantly, this framework allows both modifications that increase and decrease realism in synthetic data, which can both improve model performance. We illustrate this concept using a real-world example where data are sparse; detection of vehicles in infrared imagery. We fine-tune an initial YOLOv8 model on the ATR DSIAC infrared dataset and synthetic images generated from 3D mesh models in the Unity gaming engine, and then use XAI saliency maps to guide modification of our Unity models. We show that synthetic data can improve detection of vehicles in orientations unseen in training by 4.6% (to mAP50 = 94.6%). We further improve performance by an additional 1.5% (to 96.1%) through our new XAI-guided approach, which reduces misclassifications through both increasing and decreasing the realism of different parts of the synthetic data. Our proof-of-concept results pave the way for fine, XAI-controlled curation of synthetic datasets tailored to improve object detection performance, whilst simultaneously reducing the burden on human operators in designing and optimising these datasets.

Problem

• Optimizing synthetic data design to enhance object detection performance
• Reducing human effort in synthetic data modification using XAI
• Improving detection of underrepresented samples via XAI-guided realism adjustment

Innovation

• Uses XAI to guide synthetic data modifications
• Modifies 3D mesh models for optimal realism
• Improves detection with human-in-the-loop process