Interactive Benchmarks

2026-03-05

AI Summary

Current paradigms for evaluating reasoning capabilities are constrained by benchmark saturation, data contamination, and subjective judgments, limiting their ability to comprehensively assess model intelligence. This work proposes a novel interactive benchmarking framework that shifts the evaluation focus toward a model’s capacity to actively acquire and leverage information. By introducing a multi-turn interaction mechanism under budget constraints, the framework integrates logical reasoning, UI2HTML conversion, and mathematical tasks within contexts such as interactive proofs and games, offering a unified, objective, and contamination-resistant assessment. Experimental results demonstrate that this approach more robustly exposes significant deficiencies in contemporary models’ interactive reasoning abilities, thereby establishing a new pathway for evaluating artificial intelligence.

Abstract

Standard benchmarks have become increasingly unreliable due to saturation, subjectivity, and poor generalization. We argue that evaluating model's ability to acquire information actively is important to assess model's intelligence. We propose Interactive Benchmarks, a unified evaluation paradigm that assesses model's reasoning ability in an interactive process under budget constraints. We instantiate this framework across two settings: Interactive Proofs, where models interact with a judge to deduce objective truths or answers in logic and mathematics; and Interactive Games, where models reason strategically to maximize long-horizon utilities. Our results show that interactive benchmarks provide a robust and faithful assessment of model intelligence, revealing that there is still substantial room to improve in interactive scenarios. Project page: https://github.com/interactivebench/interactivebench

Problem

• reasoning evaluation
• interactive benchmarks
• model intelligence
• information acquisition
• objective assessment

Innovation

• Interactive Benchmarks
• reasoning evaluation
• multi-turn interaction
• strategic reasoning
• objective feedback