$S^{2}$-FracMix: Label-Preserving Self-Saliency Mixup Augmentation

2026-06-24

AI Summary

Existing mixup methods suffer from semantic distortion and high computational costs due to cross-sample interpolation, which compromises model generalization and robustness. To address this, this work proposes a label-preserving, self-saliency-based mixing augmentation framework that operates within individual images by extracting and re-embedding multi-scale salient regions. It further introduces FracMix, a novel adaptive-scale fractal structure injection mechanism, to generate structurally coherent yet challenging augmented samples. By avoiding inter-sample interference, the approach facilitates scale-invariant feature learning. Extensive experiments demonstrate state-of-the-art performance across seven benchmark tasks, including coarse- and fine-grained image classification, robustness evaluation, model calibration, object detection, and transfer learning.

Abstract

Data augmentation is known to improve generalization of deep visual models. Recent methods favor mixup strategies that generate interpolated samples to improve model performance. However, these techniques not only incur significant computational overhead, they also lead to semantic disruption of augmentation data due to cross-sample mixing. We first propose Self-Saliency ($S^2$) Mixup, which constructs challenging yet label-consistent samples by extracting multi-scale salient patches and reinserting them into non-salient regions of the same image. This promotes scale-invariant feature learning while avoiding cross-sample interference. To further enhance model robustness, we introduce FracMix, a mixing scheme that injects self-similarity patterns into salient regions using adaptive ratios. Collectively, our unified framework, $S^{2}$-FracMix, enables simultaneous learning from fractal and non-fractal structures within a single image, yielding a targeted and structurally coherent augmentation strategy. We theoretically analyze the advantage of our technique, and empirically establish its superiority over the existing methods by achieving state-of-the-art performance in extensive evaluation with seven benchmarks across classification (coarse and fine-grained), robustness, calibration, object detection, and transfer learning tasks. Project page is available at \href{https://fracmix-data-augmentation.github.io/}{fracmix-data-augmentation.github.io}

Problem

• data augmentation
• mixup
• semantic disruption
• computational overhead
• label consistency

Innovation

• Self-Saliency Mixup
• FracMix
• label-preserving augmentation
• fractal structure
• scale-invariant feature learning