Revisiting Efficient Semantic Segmentation: Learning Offsets for Better Spatial and Class Feature Alignment

2025-08-12

AI Summary

Existing lightweight semantic segmentation models suffer from misalignment between category representations and image feature spaces, stemming from the pixel-wise classification paradigm’s assumption of invariant intra-class pixel features. This misalignment is particularly detrimental in resource-constrained models. Method: We first identify and characterize this misalignment as a critical bottleneck. To address it, we propose a plug-and-play dual-branch offset learning paradigm: (i) coupled feature offset to calibrate spatial response localization, and (ii) class offset to dynamically refine class-center representations—all without modifying the backbone architecture. Contribution/Results: Our method introduces only 0.1–0.2M parameters and achieves consistent performance gains across four benchmarks, including ADE20K. On SegFormer-B0, it boosts mIoU by up to 2.7%, establishing a novel fine-grained alignment paradigm for efficient semantic segmentation.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Semantic segmentation is fundamental to vision systems requiring pixel-level scene understanding, yet deploying it on resource-constrained devices demands efficient architectures. Although existing methods achieve real-time inference through lightweight designs, we reveal their inherent limitation: misalignment between class representations and image features caused by a per-pixel classification paradigm. With experimental analysis, we find that this paradigm results in a highly challenging assumption for efficient scenarios: Image pixel features should not vary for the same category in different images. To address this dilemma, we propose a coupled dual-branch offset learning paradigm that explicitly learns feature and class offsets to dynamically refine both class representations and spatial image features. Based on the proposed paradigm, we construct an efficient semantic segmentation network, OffSeg. Notably, the offset learning paradigm can be adopted to existing methods with no additional architectural changes. Extensive experiments on four datasets, including ADE20K, Cityscapes, COCO-Stuff-164K, and Pascal Context, demonstrate consistent improvements with negligible parameters. For instance, on the ADE20K dataset, our proposed offset learning paradigm improves SegFormer-B0, SegNeXt-T, and Mask2Former-Tiny by 2.7%, 1.9%, and 2.6% mIoU, respectively, with only 0.1-0.2M additional parameters required.

Problem

• Addresses misalignment between class representations and image features
• Proposes dual-branch offset learning for dynamic feature refinement
• Enhances efficient semantic segmentation with minimal parameter overhead

Innovation

• Dual-branch offset learning for feature alignment
• Dynamic refinement of class representations
• Compatible with existing architectures