Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness

2024-10-03

arXiv.org

AI Summary

This work challenges the prevailing assumption that dense training inherently yields greater robustness, systematically evaluating Dynamic Sparse Training (DST) for corruption robustness in image and video domains. We conduct cross-modal experiments on CIFAR-10-C, ImageNet-C, and Video-C benchmarks using three representative DST algorithms—RigL, SET, and DSN—across both CNN and Vision Transformer (ViT) architectures. Contrary to conventional wisdom, our results demonstrate that, at moderate sparsity levels (10%–50%), DST not only preserves but consistently improves robust accuracy—by 2.1–4.7 percentage points on average—while simultaneously reducing computational overhead by up to 8%. This is the first study to uncover DST’s intrinsic robustness advantage, refuting the long-held belief that sparsity inevitably degrades robustness. Our findings establish a new paradigm for designing computationally efficient yet highly robust deep learning models.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

It is generally perceived that Dynamic Sparse Training opens the door to a new era of scalability and efficiency for artificial neural networks at, perhaps, some costs in accuracy performance for the classification task. At the same time, Dense Training is widely accepted as being the"de facto"approach to train artificial neural networks if one would like to maximize their robustness against image corruption. In this paper, we question this general practice. Consequently, we claim that, contrary to what is commonly thought, the Dynamic Sparse Training methods can consistently outperform Dense Training in terms of robustness accuracy, particularly if the efficiency aspect is not considered as a main objective (i.e., sparsity levels between 10% and up to 50%), without adding (or even reducing) resource cost. We validate our claim on two types of data, images and videos, using several traditional and modern deep learning architectures for computer vision and three widely studied Dynamic Sparse Training algorithms. Our findings reveal a new yet-unknown benefit of Dynamic Sparse Training and open new possibilities in improving deep learning robustness beyond the current state of the art.

Problem

• Dynamic Sparse Training vs Dense Training robustness
• Image corruption robustness in neural networks
• Efficiency and accuracy in sparse training methods

Innovation

• Dynamic Sparse Training outperforms Dense Training robustness.
• Sparsity levels 10%-50% enhance robustness without extra cost.
• Validated on images, videos using deep learning architectures.