LUMA: Benchmarking Segmentation via a Lightweight Universal Mask Adapter

2026-07-01

AI Summary

This work addresses the challenge of fairly evaluating the true performance of diverse backbone architectures in image segmentation, which has been hindered by inconsistencies in decoders, training strategies, and pretraining protocols. To this end, the authors propose LUMA (Lightweight Universal Mask Adapter), a lightweight and architecture-agnostic decoder based on cross-attention, enabling unified training recipes across backbones. Using LUMA, they conduct the first systematic benchmark under identical conditions, evaluating 20 backbones—including ViTs, CNNs, and MoEs—across 11 pretraining strategies and multiple resolutions. Their experiments reveal that the choice of pretraining objective exerts a far greater influence on segmentation performance than architectural design. Moreover, LUMA matches the accuracy of the state-of-the-art efficient ViT-based segmenter EoMT at lower computational cost and demonstrates that so-called “efficient” token mixers offer no advantage at high resolutions, with standard ViTs consistently occupying the Pareto frontier in throughput–accuracy trade-offs.

Abstract

Comparing transformer backbones for image segmentation is confounded: each is paired with a different decoder, recipe, and pretraining, so reported differences rarely reflect the backbone itself. We introduce the Lightweight Universal Mask Adapter (LUMA), a lightweight, backbone-agnostic mask-transformer head that treats any backbone as a black-box feature extractor, letting a set of queries read from its features through cheap cross-attention. LUMA matches the accuracy of EoMT, the state-of-the-art efficient ViT-segmenter, at lower cost, while attaching unchanged to isotropic, hierarchical, convolutional, and mixture-of-experts backbones alike. Holding this head fixed, we benchmark 20 backbones, 11 pretraining schemes and a range of resolutions on ADE20K and Cityscapes under one modern recipe. We find that ``efficient'' token mixers fail to deliver efficiency even at the high resolutions that motivate them, with plain ViT holding the throughput Pareto-front at every resolution. Additionally, the pretraining objective, not the architecture, the lever the field has tuned hardest, governs segmentation quality.

Problem

• image segmentation
• transformer backbone
• benchmarking
• pretraining
• model comparison

Innovation

• LUMA
• backbone-agnostic
• mask adapter
• cross-attention
• segmentation benchmarking