Photorealistic Inpainting for Perturbation-based Explanations in Ecological Monitoring

2025-09-30

AI Summary

Ecological monitoring suffers from the lack of interpretability in vision model predictions, hindering their trustworthy deployment in field settings. To address this, we propose a perturbation-based explanation method grounded in diffusion-based image inpainting, generating in-distribution, photorealistic counterfactual images via semantically coherent local edits—such as object replacement and background reset—thereby avoiding out-of-distribution artifacts induced by conventional occlusion-based approaches. Our method integrates YOLOv9 detection, optimized SAM masks, and a conditional diffusion model to localize fine-grained morphological cues (e.g., flipper texture, head轮廓) in drone-captured seal imagery from Glacier Bay. Experiments demonstrate significant improvement in expert trust in model decisions; flip-rate and confidence-drop metrics validate attribution reliability. This work establishes a new paradigm for ecological AI: auditable, human-interpretable, and transparent model reasoning.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transparent, Interpretable, Explainable ML
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Ecological monitoring is increasingly automated by vision models, yet opaque predictions limit trust and field adoption. We present an inpainting-guided, perturbation-based explanation technique that produces photorealistic, mask-localized edits that preserve scene context. Unlike masking or blurring, these edits stay in-distribution and reveal which fine-grained morphological cues drive predictions in tasks such as species recognition and trait attribution. We demonstrate the approach on a YOLOv9 detector fine-tuned for harbor seal detection in Glacier Bay drone imagery, using Segment-Anything-Model-refined masks to support two interventions: (i) object removal/replacement (e.g., replacing seals with plausible ice/water or boats) and (ii) background replacement with original animals composited onto new scenes. Explanations are assessed by re-scoring perturbed images (flip rate, confidence drop) and by expert review for ecological plausibility and interpretability. The resulting explanations localize diagnostic structures, avoid deletion artifacts common to traditional perturbations, and yield domain-relevant insights that support expert validation and more trustworthy deployment of AI in ecology.

Problem

• Explaining opaque vision model predictions in ecological monitoring
• Generating photorealistic inpainting for perturbation-based explanations
• Preserving scene context while revealing fine-grained morphological cues

Innovation

• Inpainting-guided perturbation produces photorealistic mask-localized edits
• Uses SAM-refined masks for object removal and background replacement
• Preserves scene context while revealing fine-grained morphological prediction cues