GAMBIT: A Three-Mode Benchmark for Adversarial Robustness in Multi-Agent LLM Collectives

2026-05-09

AI Summary

This work addresses the limitations of existing research, which predominantly focuses on shallow tasks and overlooks deceptive agents capable of adaptive evolution to evade detection. To bridge this gap, the authors propose GAMBIT—the first multi-agent benchmark enabling co-evolution of attack and defense strategies—grounded in the deep reasoning context of chess and augmented with an evolutionary algorithm-based framework for adaptive deception. The study introduces a comprehensive dataset comprising 27,804 annotated samples and 240 co-evolutionary strategies, demonstrates the misleading nature of zero-shot evaluation in adaptive adversarial settings (achieving only 50.5% F1), and presents a meta-learning-based rapid recalibration mechanism that yields an 8× performance gain under few-shot conditions while accelerating convergence by 20×.

Abstract

In multi-agent systems (MAS), a single deceptive agent can nullify all gains of an agentic AI collective and evade deployed defenses. However, existing adversarial studies on MAS target only shallow tasks and do not consider adaptive adversaries, which evolve their strategies to evade the very detectors trained to catch them. To address that gap, we introduce GAMBIT, a benchmark with three evaluation modes and two independent scores for evaluating imposter detectors: the first two modes measure zero-shot detection under increasing distribution shift, and a third recalibration mode measures how quickly a detector adapts to novel attacks from just 20 labeled examples. The benchmark comes with a dataset of 27,804 labeled instances spanning 240 co-evolved imposter strategies. Our contributions are threefold: (1) Using chess as a substrate deep reasoning problem and Gemini 3.1 Pro for agents, we release GAMBIT and its dataset to evaluate imposter detectors under realistic constraints against a stealthy adaptive imposter; (2) We introduce an adaptive imposter agent based on an efficient evolutionary framework, generalizable beyond chess, that collapses collective task performance while remaining essentially undetectable (50.5% F1-score with a Gemini-based detector); (3) We show that zero-shot evaluation can be highly misleading for adaptive adversaries: two detectors with near-identical zero-shot scores differ by 8x on few-shot adaptation, while the meta-learned variant converges 20x faster, a gap only visible in the recalibration mode. Altogether, GAMBIT provides the first multi-agent benchmark where adversarial attacks and defenses co-evolve, with an imposter framework generalizable beyond our use case, and promising techniques for fast recalibration in a rapidly evolving adversarial system. Code and data: https://anonymous.4open.science/r/gambit.

Problem

• adversarial robustness
• multi-agent systems
• adaptive adversaries
• imposter detection
• LLM collectives

Innovation

• adversarial robustness
• multi-agent systems
• adaptive imposter
• few-shot recalibration
• co-evolutionary benchmark