GPTNT: Benchmarking Real-Time Collaboration Between Multimodal Agents on Keep Talking And Nobody Explodes

2026-06-26

AI Summary

Existing evaluation frameworks struggle to assess the collaborative capabilities of multimodal agents in realistic scenarios characterized by time pressure, information asymmetry, and imperfect communication. To address this gap, this work introduces the first multimodal collaboration benchmark based on the cooperative puzzle game *Keep Talking and Nobody Explodes*, featuring real-time asynchronous communication: one agent observes the bomb but lacks the manual, while the other possesses the manual but cannot see the bomb, necessitating efficient coordination to defuse the device. Integrating procedurally generated puzzles, controlled experimental design, and a dynamic memory isolation mechanism, the benchmark enables comprehensive evaluation of real-time multimodal collaboration within an authentic game environment and is extensible as models evolve. Experiments reveal that current state-of-the-art open- and closed-source models consistently fail to defuse bombs under real-time constraints, exposing critical deficiencies in state tracking, time-sensitive decision-making, ambiguity resolution, and error recovery.

Abstract

Multimodal models are increasingly deployed to solve tasks collaboratively with humans or other artificial agents. Existing benchmarks show that these models possess many of the required component capabilities, but the conditions that coincide in collaboration, including time pressure, information asymmetry, and imperfect communication, are usually studied in isolation. We introduce GPTNT, a benchmark built on the cooperative video game Keep Talking and Nobody Explodes, in which two agents must coordinate to defuse procedurally generated bomb puzzles against a live countdown. One agent can see and manipulate the bomb but does not have the defusal instructions; the other has the instructions but cannot see or manipulate the bomb. Neither agent can succeed alone: success requires effective and efficient communication. Unlike turn-based proxies, GPTNT requires agents to act asynchronously and communicate in real time. GPTNT is designed to separate collaboration from reliance on memorized solutions: the instruction manual, the partner, or both can be withheld to isolate what a model derives in the moment from what it already knows. We show that GPTNT poses a substantial challenge for state-of-the-art systems: none of the closed- or open-source models we test defuses a single bomb in real time, a bar that human players clear. Through controlled experiments, we identify critical weaknesses in state tracking, efficient action under time pressure, ambiguity handling, and error recovery. We release GPTNT as a benchmark for collaborative performance that current evaluations leave unmeasured. Because it runs on the real game, GPTNT benefits from procedural generation and inherits a living modding community, allowing the benchmark to evolve as models improve rather than being solved once and retired.

Problem

• real-time collaboration
• multimodal agents
• information asymmetry
• time pressure
• imperfect communication

Innovation

• real-time collaboration
• multimodal agents
• information asymmetry
• procedural generation
• asynchronous communication