When Does Resolution Help a Frozen Backbone? Global Attention at Resolution Predicts Scalable Adaptation for Camouflaged and Marine Animal Segmentation

2026-07-02

AI Summary

This work addresses the limitation of frozen vision foundation models in fine-grained segmentation, where inappropriate backbone selection prevents high-resolution inputs from improving accuracy. The study identifies global attention mechanisms as pivotal for achieving resolution-scalable adaptation and introduces a unified low-rank adaptation framework based on this insight. Through controlled experiments across six backbone architectures, the authors validate their design principles via end-to-end training incorporating Side-stem and Attention-gated U-Net structures with single-channel RGB inputs. The proposed method achieves state-of-the-art S-measure performance on four camouflaged object segmentation benchmarks and leads across all marine and salient object segmentation datasets, notably setting a new record on MAS3K with an mIoU of 0.878.

Abstract

Adapting frozen vision foundation models to fine-grained segmentation now largely depends on backbone selection. Whether the backbone applies global attention to a high-resolution token set predicts whether a low-rank adapter turns resolution into accuracy. Isotropic ViTs attend globally over the full grid and keep improving with resolution; hierarchical backbones confine early attention to local windows and pool the grid before their global stages, plateauing at lower resolutions. A controlled six-backbone study establishes the pattern, and editing the backbone points to the cause: pooling keeps the benefit, removing global attention does not. The effect is specific to low-rank adaptation. Under one fixed pipeline, SALT (Side-stem, Attention-gated U-Net, Low-rank Tuning), one RGB-only pass on a strong isotropic backbone wins the best S-measure on the four data-matched camouflaged sets, and leads every marine and salient set. It reaches a new state of the art on both marine-animal benchmarks (MAS3K mIoU 0.878).

Problem

• frozen backbone
• global attention
• resolution
• low-rank adaptation
• fine-grained segmentation

Innovation

• global attention
• resolution scaling
• low-rank adaptation
• frozen backbone
• isotropic ViT