StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns

2026-06-17

AI Summary

Existing evaluation metrics struggle to assess the robustness of code agents in prolonged, multi-turn interactive programming scenarios. To address this gap, this work proposes the first black-box, language-agnostic benchmark centered on consecutive interaction rounds, driving agents to iteratively develop a REST API service through 100 programmatically generated change requests. The platform ensures reproducibility and realism by employing an isolated HTTP execution environment, a structured action space, programmatic testing, and a hybrid change sampler that emulates authentic “ambient programming” conditions. Experimental results reveal that all models fail within 5–6 rounds; however, incorporating a feedback-based retry mechanism improves success duration by up to 12×. Furthermore, high-performing agents exhibit significant sensitivity to the evaluation framework, with performance varying by as much as 6× between optimal and suboptimal configurations.

Abstract

We introduce StaminaBench, a benchmark that measures the stamina of coding agents: how many consecutive interaction turns (change requests) they can handle before failing. Unlike the prevailing fraction-of-tasks-solved metric, this matches real vibe-coding where sessions run dozens or hundreds of turns. In StaminaBench, agents implement a REST API server and modify it across a tunable number of procedurally generated follow-up change requests - 100 in our experiments, resulting in codebases of up to 6,000 lines. Tests are generated fully programmatically without LLM involvement, ensuring reproducibility and reliability; change sequences are drawn from either a hardcoded or LLM-driven sampler, both constrained to a structured action space to ensure changes are valid. The agent and the server run in an isolated environment and communicate with the benchmark through HTTP, making testing fully black-box and language-agnostic. We evaluate six agent harnesses paired with seven open-source LLMs across 20 scenarios of 100 turns each and find that: (1) all the tested models fail within 5-6 turns, confirming that vibe-coding-style programming without thorough testing produces bugs; (2) passing test feedback back to the agent and allowing it to retry improves passed turn count by up to 12x; and (3) a good harness is required for strong performance: stronger models exhibit up to a 6x gap between their best and worst harness, while weaker models fail with any harness. We release the benchmark and the generated tasks to enable further research into multi-turn coding agent behavior. Benchmark code and data: github.com/amazon-science/StaminaBench.

Problem

• coding agents
• multi-turn interaction
• stamina benchmarking
• REST API development
• long-horizon coding

Innovation

• StaminaBench
• multi-turn coding
• black-box evaluation
• procedurally generated tasks
• code agent stamina