Breaking Self-Attention Failure: Rethinking Query Initialization for Infrared Small Target Detection

2026-01-06

arXiv.org

AI Summary

This work addresses the vulnerability of self-attention mechanisms in infrared small target detection, where strong background clutter compromises query initialization reliability and degrades localization accuracy. The study is the first to elucidate this failure mechanism and introduces SEF-DETR, a novel framework that establishes a joint frequency-spatial paradigm for query initialization. SEF-DETR integrates Frequency-guided Proposal Selection (FPS), Dynamic Embedding Enhancement (DEE), and Reliability-consistent Fusion (RCF) to refine query generation within an end-to-end DETR architecture. Extensive experiments demonstrate that SEF-DETR significantly outperforms state-of-the-art methods across three public infrared small target datasets, achieving more robust and efficient detection performance.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Infrared small target detection (IRSTD) faces significant challenges due to the low signal-to-noise ratio (SNR), small target size, and complex cluttered backgrounds. Although recent DETR-based detectors benefit from global context modeling, they exhibit notable performance degradation on IRSTD. We revisit this phenomenon and reveal that the target-relevant embeddings of IRST are inevitably overwhelmed by dominant background features due to the self-attention mechanism, leading to unreliable query initialization and inaccurate target localization. To address this issue, we propose SEF-DETR, a novel framework that refines query initialization for IRSTD. Specifically, SEF-DETR consists of three components: Frequency-guided Patch Screening (FPS), Dynamic Embedding Enhancement (DEE), and Reliability-Consistency-aware Fusion (RCF). The FPS module leverages the Fourier spectrum of local patches to construct a target-relevant density map, suppressing background-dominated features. DEE strengthens multi-scale representations in a target-aware manner, while RCF further refines object queries by enforcing spatial-frequency consistency and reliability. Extensive experiments on three public IRSTD datasets demonstrate that SEF-DETR achieves superior detection performance compared to state-of-the-art methods, delivering a robust and efficient solution for infrared small target detection task.

Problem

• Infrared small target detection
• Self-attention failure
• Query initialization
• Low signal-to-noise ratio
• Background clutter

Innovation

• SEF-DETR
• query initialization
• infrared small target detection
• self-attention failure
• frequency-guided patch screening