Is Adversarial Training with Compressed Datasets Effective?

2024-02-08

arXiv.org

AI Summary

This work investigates whether dataset compression can effectively support adversarial training. It identifies a critical limitation: existing compression methods produce synthetic datasets that fail to transfer adversarial robustness. To address this, the authors propose the first robustness-aware dataset compression framework, grounded in Minimum Finite Cover (MFC), which explicitly incorporates adversarial robustness into the compression objective. Theoretically, the MFC-based method is provably robust, computed in a single pass, model-agnostic, and generalizable. Through rigorous comparison—leveraging generalized adversarial loss minimization and distribution matching—the framework significantly outperforms state-of-the-art dataset compression (DC) methods across three benchmark datasets, achieving superior trade-offs between clean accuracy and adversarial robustness. Moreover, adversarial training on MFC-compressed datasets converges faster and yields higher final robustness than training on datasets compressed via competing DC approaches.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Data Mining & Knowledge Management: Data Compression

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Dataset Condensation (DC) refers to the recent class of dataset compression methods that generate a smaller, synthetic, dataset from a larger dataset. This synthetic dataset retains the essential information of the original dataset, enabling models trained on it to achieve performance levels comparable to those trained on the full dataset. Most current DC methods have mainly concerned with achieving high test performance with limited data budget, and have not directly addressed the question of adversarial robustness. In this work, we investigate the impact of adversarial robustness on models trained with compressed datasets. We show that the compressed datasets obtained from DC methods are not effective in transferring adversarial robustness to models. As a solution to improve dataset compression efficiency and adversarial robustness simultaneously, we propose a novel robustness-aware dataset compression method based on finding the Minimal Finite Covering (MFC) of the dataset. The proposed method is (1) obtained by one-time computation and is applicable for any model, (2) more effective than DC methods when applying adversarial training over MFC, (3) provably robust by minimizing the generalized adversarial loss. Additionally, empirical evaluation on three datasets shows that the proposed method is able to achieve better robustness and performance trade-off compared to DC methods such as distribution matching.

Problem

• Evaluating adversarial robustness in dataset-compressed models
• Addressing inefficiency of DC methods in robustness transfer
• Proposing MFC-based compression for improved adversarial resilience

Innovation

• Robustness-aware dataset compression via Minimal Finite Covering
• Minimizes generalized adversarial loss provably
• One-time computation applicable for any model