Rethinking the Role of Spatial Mixing

2025-03-21

AI Summary

This work investigates the fundamental role and necessity of spatial mixing operations in visual models. We conduct systematic ablation studies on ResNet and ConvMixer architectures, complemented by PGD-based adversarial evaluation and pixel-wise random shuffling reconstruction tests. Our findings reveal: (1) spatial mixing can be drastically simplified—retaining only random initialization or even full parameter freezing—while preserving over 98% of original ImageNet classification accuracy; (2) such simplification not only preserves performance but substantially enhances adversarial robustness (marked improvement in PGD attack accuracy) and structural recovery capability (successful reconstruction of severely shuffled pixel images). Crucially, we demonstrate for the first time that the core value of spatial mixing lies not in learning complex transformations, but in providing a lightweight, robust, and interpretable implicit structural prior—challenging prevailing assumptions about the necessity of learned spatial aggregation in vision models.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Until quite recently, the backbone of nearly every state-of-the-art computer vision model has been the 2D convolution. At its core, a 2D convolution simultaneously mixes information across both the spatial and channel dimensions of a representation. Many recent computer vision architectures consist of sequences of isotropic blocks that disentangle the spatial and channel-mixing components. This separation of the operations allows us to more closely juxtapose the effects of spatial and channel mixing in deep learning. In this paper, we take an initial step towards garnering a deeper understanding of the roles of these mixing operations. Through our experiments and analysis, we discover that on both classical (ResNet) and cutting-edge (ConvMixer) models, we can reach nearly the same level of classification performance by and leaving the spatial mixers at their random initializations. Furthermore, we show that models with random, fixed spatial mixing are naturally more robust to adversarial perturbations. Lastly, we show that this phenomenon extends past the classification regime, as such models can also decode pixel-shuffled images.

Problem

• Understanding roles of spatial and channel mixing in deep learning
• Evaluating performance of models with random spatial mixers
• Assessing robustness of fixed spatial mixing to adversarial attacks

Innovation

• Separates spatial and channel mixing operations
• Uses random fixed spatial mixers for robustness
• Decodes pixel-shuffled images effectively