Average Certified Radius is a Poor Metric for Randomized Smoothing

2024-10-09

arXiv.org

AI Summary

Average Certified Radius (ACR) is fundamentally flawed as a robustness metric for randomized smoothing, as it is easily manipulated by trivial classifiers, overly biased toward easy samples, and fails to reflect true robustness on hard examples. Method: We conduct theoretical analysis—deriving lower bounds and proving ACR’s sensitivity to sample difficulty—and systematic experiments—including multi-strategy comparisons, reweighted sampling, and selective optimization—to rigorously examine ACR’s validity. Contribution/Results: We are the first to theoretically and empirically refute ACR’s legitimacy as a robustness measure. We further propose non-robust strategies—sample discarding, radius weighting, and extreme optimization of easy samples—that achieve state-of-the-art ACR on CIFAR-10 and ImageNet *without any robust training*, while drastically degrading robustness on hard samples. Our findings expose severe misleadingness in mainstream ACR-improvement methods and fundamentally undermine ACR’s validity as an evaluation standard.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Randomized smoothing is a popular approach for providing certified robustness guarantees against adversarial attacks, and has become an active area of research. Over the past years, the average certified radius (ACR) has emerged as the most important metric for comparing methods and tracking progress in the field. However, in this work, for the first time we show that ACR is a poor metric for evaluating robustness guarantees provided by randomized smoothing. We theoretically prove not only that a trivial classifier can have arbitrarily large ACR, but also that ACR is much more sensitive to improvements on easy samples than on hard ones. Empirically, we confirm that existing training strategies, though improving ACR with different approaches, reduce the model's robustness on hard samples consistently. To strengthen our conclusion, we propose strategies, including explicitly discarding hard samples, reweighting the dataset with approximate certified radius, and extreme optimization for easy samples, to achieve state-of-the-art ACR, without training for robustness on the full data distribution. Overall, our results suggest that ACR has introduced a strong undesired bias to the field, and its application should be discontinued when evaluating randomized smoothing.

Problem

• Randomized Smoothing
• Adversarial Attacks
• Average Certified Radius

Innovation

• Average Certification Radius
• Randomized Smoothing
• Reliability Enhancement