FastAT Benchmark: A Comprehensive Framework for Fair Evaluation of Fast Adversarial Training Methods

2026-04-22

AI Summary

This work addresses the lack of standardized evaluation in existing fast adversarial training (FastAT) methods, which hinders fair performance comparison due to confounding factors such as model architectures, training configurations, and external data usage. To remedy this, we propose a principled benchmarking framework grounded in three core tenets: unified model architecture, standardized training protocols, and exclusion of external data. Within a single codebase, we implement over twenty FastAT methods and establish reproducible baselines on CIFAR-10, CIFAR-100, and Tiny-ImageNet. Robustness is comprehensively evaluated using PGD, AutoAttack, and CR Attack, while GPU training time and peak memory consumption are meticulously recorded. Our experiments systematically demonstrate—for the first time—that certain single-step methods achieve comparable or superior robustness to PGD-AT at substantially lower computational cost, with no single approach dominating across all metrics, thereby underscoring the critical need for fair and consistent evaluation. The full benchmark is publicly released.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements

Abstract

Fast Adversarial Training (FastAT) seeks to achieve adversarial robustness at a fraction of the computational cost incurred by standard multi-step methods such as PGD-AT. Although numerous FastAT techniques have been proposed in recent years, fair comparison among them remains elusive. Existing benchmarks and public leaderboards typically permit diverse model architectures, varying training configurations, and external data sources, making it unclear whether reported improvements reflect genuine algorithmic advances or merely more favorable experimental conditions. To address this problem, we introduce the FastAT Benchmark, a controlled evaluation framework built on three core design principles: unified architecture requirements, standardized training settings, and strict prohibition of external or synthetic data. The benchmark implements over twenty representative FastAT methods within a single codebase, enabling direct and reproducible comparison. Each method is assessed through a dual-metric evaluation framework that measures both adversarial robustness (accuracy under PGD, AutoAttack, and CR Attack) and computational cost (GPU training time and peak memory footprint). Comprehensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet provide reliable baseline measurements and reveal that well-designed single-step methods can match or surpass PGD-AT robustness at substantially lower cost, while no single method dominates across all evaluation dimensions. The complete benchmark, including source code, configuration files, and experimental results, is publicly available to support transparent and fair evaluation of future FastAT research.

Problem

• Fast Adversarial Training
• fair evaluation
• benchmark
• adversarial robustness
• standardized comparison

Innovation

• Fast Adversarial Training
• Benchmark
• Fair Evaluation
• Standardized Training
• Adversarial Robustness