Efficient Semantic Segmentation via Lightweight Multiple-Information Interaction Network

2024-10-03

arXiv.org

AI Summary

To address the high computational cost and memory footprint in real-time semantic segmentation, this paper proposes the Lightweight Multi-Information Interaction Network (LMIINet). Our method introduces three key innovations: (1) a novel Lightweight Feature Interaction Bottleneck (LFIB) module for efficient cross-layer feature fusion; (2) an enhanced Flatten Transformer architecture to strengthen joint local-global contextual modeling; and (3) a learnable coefficient mechanism for adaptive multi-scale feature weighting. Evaluated on Cityscapes and CamVid, LMIINet achieves 72.0% mIoU at 100 FPS and 69.94% mIoU at 160 FPS, respectively, with only 0.72M parameters and 11.74G FLOPs (on RTX 2080 Ti). These results significantly outperform existing lightweight approaches in both accuracy and efficiency, establishing a new trade-off frontier for real-time semantic segmentation.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Recently, integrating the local modeling capabilities of Convolutional Neural Networks (CNNs) with the global dependency strengths of Transformers has created a sensation in the semantic segmentation community. However, substantial computational workloads and high hardware memory demands remain major obstacles to their further application in real-time scenarios. In this work, we propose a Lightweight Multiple-Information Interaction Network (LMIINet) for real-time semantic segmentation, which effectively combines CNNs and Transformers while reducing redundant computations and memory footprints. It features Lightweight Feature Interaction Bottleneck (LFIB) modules comprising efficient convolutions that enhance context integration. Additionally, improvements are made to the Flatten Transformer by enhancing local and global feature interaction to capture detailed semantic information. Incorporating a combination coefficient learning scheme in both LFIB and Transformer blocks facilitates improved feature interaction. Extensive experiments demonstrate that LMIINet excels in balancing accuracy and efficiency. With only 0.72M parameters and 11.74G FLOPs (Floating Point Operations Per Second), LMIINet achieves 72.0% mIoU at 100 FPS (Frames Per Second) on the Cityscapes test set and 69.94% mIoU (mean Intersection over Union) at 160 FPS on the CamVid test dataset using a single RTX2080Ti GPU.

Problem

• Lightweight network for real-time semantic segmentation
• Reducing computational workloads and memory demands
• Combining CNNs and Transformers efficiently

Innovation

• Lightweight Multiple-Information Interaction Network
• Lightweight Feature Interaction Bottleneck modules
• Improved Flatten Transformer for feature interaction