Combining Machine Learning Defenses without Conflicts

2024-11-14

arXiv.org

AI Summary

When machine learning models must simultaneously withstand security, privacy, and fairness threats, existing defense combinations often suffer from methodological conflicts that degrade efficacy and incur prohibitive empirical validation costs. To address this, we propose DefCon—the first framework for assessing the effectiveness of defense combinations while satisfying four critical desiderata: accuracy, scalability, non-intrusiveness, and generality. DefCon leverages defense interaction modeling and lightweight meta-feature evaluation, enabling principle-driven compositional analysis without modifying original defense modules. It achieves 90% accuracy on eight established combinations and 81% on thirty novel ones, substantially reducing experimental overhead. Its core innovation lies in transcending heuristic, experience-based composition paradigms by establishing an interpretable, reusable theoretical criterion for combination validity.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Machine learning (ML) defenses protect against various risks to security, privacy, and fairness. Real-life models need simultaneous protection against multiple different risks which necessitates combining multiple defenses. But combining defenses with conflicting interactions in an ML model can be ineffective, incurring a significant drop in the effectiveness of one or more defenses being combined. Practitioners need a way to determine if a given combination can be effective. Experimentally identifying effective combinations can be time-consuming and expensive, particularly when multiple defenses need to be combined. We need an inexpensive, easy-to-use combination technique to identify effective combinations. Ideally, a combination technique should be (a) accurate (correctly identifies whether a combination is effective or not), (b) scalable (allows combining multiple defenses), (c) non-invasive (requires no change to the defenses being combined), and (d) general (is applicable to different types of defenses). Prior works have identified several ad-hoc techniques but none satisfy all the requirements above. We propose a principled combination technique, DefCon, to identify effective defense combinations. DefCon meets all requirements, achieving 90% accuracy on eight combinations explored in prior work and 81% in 30 previously unexplored combinations that we empirically evaluate in this paper.

Problem

• Identify effective combinations of ML defenses without conflicts
• Ensure combined defenses maintain security, privacy, and fairness
• Provide scalable, accurate, and non-invasive defense combination technique

Innovation

• Combines ML defenses without conflicts
• Uses DefCon for effective combinations
• Ensures accuracy, scalability, non-invasiveness, generality