🤖 AI Summary
Real-world black-box AI systems lack verifiable robustness against decision-based attacks. To address this, we propose a co-evolutionary multi-agent game framework that— for the first time—generalizes “adaptivity” into a bidirectional, dynamic attack-defense evolution mechanism. Leveraging model-free reinforcement learning, our approach jointly optimizes adaptive attack policies and proactive defense responses, enabling closed-loop adversarial optimization in black-box settings without access to model internals or gradients. This breaks from conventional unidirectional robustness verification paradigms and establishes the first robustness evaluation framework grounded in co-evolutionary principles. Extensive experiments on real-world ML systems demonstrate that our method achieves significantly higher assessment reliability than existing state-of-the-art approaches. Empirically, it reveals adaptive adversarial interaction—not static perturbation resistance—as the core challenge in black-box AI security.
📝 Abstract
Despite considerable efforts on making them robust, real-world AI-based systems remain vulnerable to decision based attacks, as definitive proofs of their operational robustness have so far proven intractable. Canonical robustness evaluation relies on adaptive attacks, which leverage complete knowledge of the defense and are tailored to bypass it. This work broadens the notion of adaptivity, which we employ to enhance both attacks and defenses, showing how they can benefit from mutual learning through interaction. We introduce a framework for adaptively optimizing black-box attacks and defenses under the competitive game they form. To assess robustness reliably, it is essential to evaluate against realistic and worst-case attacks. We thus enhance attacks and their evasive arsenal together using RL, apply the same principle to defenses, and evaluate them first independently and then jointly under a multi-agent perspective. We find that active defenses, those that dynamically control system responses, are an essential complement to model hardening against decision-based attacks; that these defenses can be circumvented by adaptive attacks, something that elicits defenses being adaptive too. Our findings, supported by an extensive theoretical and empirical investigation, confirm that adaptive adversaries pose a serious threat to black-box AI-based systems, rekindling the proverbial arms race. Notably, our approach outperforms the state-of-the-art black-box attacks and defenses, while bringing them together to render effective insights into the robustness of real-world deployed ML-based systems.
