Revitalizing Dense Material Segmentation: Stabilized Vision Transformers and the Generalization Paradox

2026-05-22

AI Summary

This work addresses the performance bottleneck in dense material segmentation caused by high-variance gradients and data split bias, which limits the generalization of existing methods on amorphous textures and real-world scenes. To overcome these challenges, the authors propose a stable training scheme that integrates high-fidelity logit projection, query entropy regularization, and physics-compliant augmentation, significantly enhancing Vision Transformer performance. Built upon SegFormer and Mask2Former architectures, the optimized SegFormer-B5 achieves a new state-of-the-art mIoU of 0.4572 on the original Apple-DMS test set. Although higher performance (0.5276 mIoU) is observed on a re-partitioned dataset, empirical analysis reveals a “generalization paradox”: overly permissive data splits impair out-of-distribution generalization, underscoring the critical importance of the original, stricter split for advancing physically grounded intelligence.

Abstract

Material segmentation, the pixel-wise classification of physical surface properties, remains a challenging problem in computer vision, requiring physicochemical understanding distinct from object-centric parsing. Despite the introduction of the rigorous Apple Dense Material Segmentation (DMS) dataset, the benchmark has suffered from attrition and stagnation, increasingly overshadowed by geometry-biased foundation models. In this paper, we revive the Apple-DMS benchmark to establish a modern Vision Transformer baseline. We conduct an exhaustive evaluation of SegFormer and Mask2Former architectures, revealing that standard training paradigms fail on amorphous texture fields due to high-variance gradients. To address this, we introduce a stabilized training recipe featuring High-Fidelity Logit Projection, Query Entropy Regularization, and a domain-specific, physics-compliant augmentation pipeline. Our optimized SegFormer-B5 achieves a new State-of-the-Art (SOTA) of 0.4572 mIoU on the original dataset split, significantly surpassing the prior convolutional baseline. Furthermore, we identify a critical "Generalization Paradox": while re-partitioning the dataset into a data-rich 80/10/10 split inflates the metric to 0.5276 mIoU, expert qualitative analysis reveals this induces distributional homogenization, severely degrading real-world, out-of-distribution performance. By releasing our recovered dataset index and robust training framework, we demonstrate that material perception is far from solved and urge the community to leverage the rigorous original split to drive genuine progress in physically grounded artificial intelligence.

Problem

• material segmentation
• generalization paradox
• dense material segmentation
• out-of-distribution generalization
• Vision Transformers

Innovation

• Vision Transformers
• Material Segmentation
• Stabilized Training
• Generalization Paradox
• Physics-Compliant Augmentation