Progressive Pixel-Neighborhood Deformable Cross-Attention for Multispectral Object Detection

2026-06-22

AI Summary

This work addresses key challenges in multispectral object detection—inefficient cross-modal feature alignment, high computational cost of global attention, and the inability of fixed receptive fields to model nonlinear spatial relationships—by proposing the PNAFusion framework. The method introduces Pixel Neighborhood Cross-Attention (PNCA) to eliminate redundant global matching, designs an Adaptive Deformable Alignment (ADA) module to capture nonlinear spatial mappings, and incorporates a progressive feedback mechanism to iteratively refine fusion between visible and thermal infrared features. Integrated into YOLOv5 and Co-DETR, PNAFusion achieves 84.2, 90.5, and 85.5 mAP@0.5 on the FLIR, M3FD, and DroneVehicle datasets, respectively, while reducing GPU memory consumption by 33.0% compared to ICAFusion and lowering theoretical FLOPs from 194.8G to 156.4G.

Abstract

Effective cross-modal feature alignment and interaction are central challenges in multispectral object detection. Although global cross-attention provides strong long-range modeling ability, its quadratic complexity with respect to feature size limits deployment on resource-constrained platforms. We therefore propose Progressive Pixel-Neighborhood Deformable Cross-Attention for multispectral feature fusion, termed PNAFusion. The proposed framework is motivated by two observations: weak misalignment between visible and thermal images is usually concentrated around local neighborhoods, and semantic correspondence across modalities often follows non-linear spatial mappings that fixed receptive fields cannot model well. To address these issues, PNAFusion incorporates local spatial priors into its architectural design to concentrate feature interaction and alignment on the most relevant neighborhoods. Specifically, a Pixel-Neighborhood Cross-Attention (PNCA) module is introduced to avoid redundant global feature matching and suppress background noise. Meanwhile, an Adaptive Deformable Alignment (ADA) module captures non-linear spatial correspondences through learned pixel-wise offsets. These components are further integrated through an iterative feedback mechanism to progressively refine cross-modal feature alignment. Experiments on FLIR, M3FD, and DroneVehicle show that PNAFusion achieves 84.2, 90.5, and 85.5 mAP@0.5, respectively, under the YOLOv5 detector, and further reaches 86.8 mAP@0.5 on FLIR and 90.8 mAP@0.5 on M3FD when transferred to Co-DETR. Efficiency analysis indicates that PNAFusion reduces allocated GPU memory by 33.0\% compared with ICAFusion and reduces theoretical FLOPs from 194.8 G to 156.4 G, although the deformable sampling and iterative refinement introduce additional latency. Our code will be available at https://github.com/DanielQiuTian/PNAFusion.

Problem

• multispectral object detection
• cross-modal feature alignment
• feature interaction
• computational complexity
• resource-constrained platforms

Innovation

• Deformable Cross-Attention
• Pixel-Neighborhood Attention
• Multispectral Object Detection
• Adaptive Feature Alignment
• Efficient Cross-Modal Fusion