DMAT: An End-to-End Framework for Joint Atmospheric Turbulence Mitigation and Object Detection

2025-07-06

AI Summary

Atmospheric turbulence degrades surveillance imagery through spatial and temporal distortions, severely compromising visual quality and object detection accuracy. To address this, we propose the first end-to-end joint modeling framework that unifies turbulence mitigation and object detection into a single optimization objective. Our method innovatively introduces a 3D Mamba architecture to capture long-range spatiotemporal distortions and designs a cross-level interaction mechanism between low-level image restoration and high-level semantic features. Integrated with pyramid feature extraction and end-to-end backpropagation, the framework enables co-optimization of image reconstruction and detection. Evaluated on a synthetic turbulence dataset, our approach achieves an average 15% improvement over state-of-the-art methods in both image fidelity (PSNR/SSIM) and detection performance (mAP), effectively alleviating spatiotemporal distortion while balancing reconstruction fidelity and detection robustness.

Technology Category

• Computer Vision: Object Detection & Categorization
• Search and Optimization: Learning to Search
• Machine Learning: Feature Construction/Reformulation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Atmospheric Turbulence (AT) degrades the clarity and accuracy of surveillance imagery, posing challenges not only for visualization quality but also for object classification and scene tracking. Deep learning-based methods have been proposed to improve visual quality, but spatio-temporal distortions remain a significant issue. Although deep learning-based object detection performs well under normal conditions, it struggles to operate effectively on sequences distorted by atmospheric turbulence. In this paper, we propose a novel framework that learns to compensate for distorted features while simultaneously improving visualization and object detection. This end-to-end framework leverages and exchanges knowledge of low-level distorted features in the AT mitigator with semantic features extracted in the object detector. Specifically, in the AT mitigator a 3D Mamba-based structure is used to handle the spatio-temporal displacements and blurring caused by turbulence. Features are extracted in a pyramid manner during the mitigation stage and passed to the detector. Optimization is achieved through back-propagation in both the AT mitigator and object detector. Our proposed DMAT outperforms state-of-the-art AT mitigation and object detection systems up to a 15% improvement on datasets corrupted by generated turbulence.

Problem

• Mitigates atmospheric turbulence effects on surveillance imagery
• Improves object detection in turbulence-distorted sequences
• Integrates feature compensation and semantic extraction end-to-end

Innovation

• 3D Mamba-based structure handles spatio-temporal distortions
• Pyramid feature extraction during mitigation stage
• End-to-end framework combines AT mitigation and detection