🤖 AI Summary
This work addresses the significant computational redundancy in existing lightweight semantic segmentation methods, which independently compute attention at each level of multi-scale decoders. To overcome this inefficiency, the authors propose a Cross-Stage Attention Propagation (CSAP) mechanism that computes attention only once at the deepest feature layer and efficiently propagates it to shallower layers, thereby eliminating redundant query-key operations while preserving multi-scale contextual modeling capability. Built upon CSAP, the lightweight model CSAP-Tiny achieves 42.9% mIoU on ADE20K with only 5.5 GFLOPs, outperforming SegNeXt-Tiny by 1.8% mIoU while reducing computational cost by 16.8%. This approach is the first to enable cross-stage sharing of attention distributions, effectively balancing efficiency and performance.
📝 Abstract
Recent lightweight semantic segmentation methods have made significant progress by combining compact backbones with efficient decoder heads. However, most multi-scale decoders compute attention independently at each feature scale, introducing substantial redundancy since the resulting attention distributions across scales are strongly correlated. We propose Cross-Stage Attention Propagation (CSAP), a decoder framework that computes attention at the deepest feature scale and propagates the resulting attention maps to shallower stages, bypassing query-key computation at those stages entirely. This design preserves multi-scale contextual reasoning while substantially reducing the decoder's computational cost. CSAP-Tiny achieves 42.9% mIoU on ADE20K with only 5.5 GFLOPs, 80.5% on Cityscapes with 21.5 GFLOPs, and 40.9% on COCO-Stuff 164K with 5.5 GFLOPs, surpassing SegNeXt-Tiny by +1.8% on ADE20K while requiring 16.8% fewer floating-point operations.