Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation

2026-04-15

AI Summary

This work proposes a coarse-grained, patch-based sparse mixture-of-experts (MoE) layer tailored for convolutional neural networks in semantic segmentation, achieving notable performance gains with minimal computational overhead. It introduces block-level sparse MoE to dense prediction tasks for the first time, dynamically routing local image regions to a small subset of convolutional experts. Experiments on Cityscapes and BDD100K demonstrate consistent performance improvements across multiple backbone architectures, with gains of up to +3.9 mIoU. The study further uncovers a critical relationship between the routing mechanism and expert specialization, offering empirical insights that inform the design of efficient MoE architectures for dense visual understanding tasks.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Segmentation
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Sparse mixture-of-experts (MoE) layers have been shown to substantially increase model capacity without a proportional increase in computational cost and are widely used in transformer architectures, where they typically replace feed-forward network blocks. In contrast, integrating sparse MoE layers into convolutional neural networks (CNNs) remains inconsistent, with most prior work focusing on fine-grained MoEs operating at the filter or channel levels. In this work, we investigate a coarser, patch-wise formulation of sparse MoE layers for semantic segmentation, where local regions are routed to a small subset of convolutional experts. Through experiments on the Cityscapes and BDD100K datasets using encoder-decoder and backbone-based CNNs, we conduct a design analysis to assess how architectural choices affect routing dynamics and expert specialization. Our results demonstrate consistent, architecture-dependent improvements (up to +3.9 mIoU) with little computational overhead, while revealing strong design sensitivity. Our work provides empirical insights into the design and internal dynamics of sparse MoE layers in CNN-based dense prediction. Our code is available at https://github.com/KASTEL-MobilityLab/moe-layers/.

Problem

• Sparse Mixture-of-Experts
• CNN-based Semantic Segmentation
• Patch-wise Routing
• Expert Specialization
• Dense Prediction

Innovation

• Sparse Mixture-of-Experts
• Patch-wise Routing
• CNN-based Semantic Segmentation
• Expert Specialization
• Dense Prediction