Unlocking Noise-Resistant Vision: Key Architectural Secrets for Robust Models

2025-09-25

AI Summary

This study investigates the intrinsic robustness mechanisms of visual architectures against additive Gaussian noise, aiming to uncover causal relationships between architectural design choices and noise resilience. Method: Leveraging theoretical modeling and large-scale empirical analysis across 1,174 pretrained models, we systematically attribute robustness to specific components—stem convolutional kernels, pooling strategies, and preprocessing pipelines—grounded in low-pass filtering, anti-aliasing downsampling, noise-suppressing average pooling, and pixel-space Lipschitz constant analysis. Contribution/Results: We establish an interpretable theoretical framework for noise robustness and derive plug-and-play design principles for Vision Transformers (ViTs). Our analysis identifies four universal noise-resilient architectural patterns. On ImageNet-C, these principles elevate model rankings by up to 506 positions and improve top-1 accuracy by 21.6%, empirically validating the consistency between theoretical predictions and practical performance.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Large pretrained models with web data
• Security and Privacy: Large-scale security measurements

Abstract

While the robustness of vision models is often measured, their dependence on specific architectural design choices is rarely dissected. We investigate why certain vision architectures are inherently more robust to additive Gaussian noise and convert these empirical insights into simple, actionable design rules. Specifically, we performed extensive evaluations on 1,174 pretrained vision models, empirically identifying four consistent design patterns for improved robustness against Gaussian noise: larger stem kernels, smaller input resolutions, average pooling, and supervised vision transformers (ViTs) rather than CLIP ViTs, which yield up to 506 rank improvements and 21.6%p accuracy gains. We then develop a theoretical analysis that explains these findings, converting observed correlations into causal mechanisms. First, we prove that low-pass stem kernels attenuate noise with a gain that decreases quadratically with kernel size and that anti-aliased downsampling reduces noise energy roughly in proportion to the square of the downsampling factor. Second, we demonstrate that average pooling is unbiased and suppresses noise in proportion to the pooling window area, whereas max pooling incurs a positive bias that grows slowly with window size and yields a relatively higher mean-squared error and greater worst-case sensitivity. Third, we reveal and explain the vulnerability of CLIP ViTs via a pixel-space Lipschitz bound: The smaller normalization standard deviations used in CLIP preprocessing amplify worst-case sensitivity by up to 1.91 times relative to the Inception-style preprocessing common in supervised ViTs. Our results collectively disentangle robustness into interpretable modules, provide a theory that explains the observed trends, and build practical, plug-and-play guidelines for designing vision models more robust against Gaussian noise.

Problem

• Investigating architectural design choices that enhance vision model robustness to Gaussian noise
• Identifying four key design patterns for improved noise resistance in vision architectures
• Developing theoretical explanations for how specific components affect noise sensitivity

Innovation

• Larger stem kernels and smaller input resolutions
• Average pooling instead of max pooling
• Supervised ViTs over CLIP ViTs for robustness