GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments

2026-07-03

AI Summary

Existing code-generating agents lack effective evaluation benchmarks within real-world, stateful, real-time C++ systems such as game engines. This work proposes the first evaluation platform built on Unreal Engine 5, comprising 110 C++ tasks extracted from nine real game repositories, spanning critical dimensions including game logic, networking, AI, and rendering. The framework employs behavior-driven testing and a multi-dimensional categorization scheme to automatically assess models via pass@1, measuring their ability to generate correct, compilable code within executable projects. Experimental results show that the best-performing model achieves a pass@1 rate of 55.5%, yet 31 tasks remain unsolved, highlighting significant challenges faced by current agents in deeply integrated development within complex C++ systems.

Abstract

Game engines provide real-time simulation, rendering, physics, interaction, networking, and asset pipelines, making them valuable not only for games but also for 3D applications in healthcare, robotics, architecture, manufacturing, and related domains. Because game development is where these systems are most mature and publicly available, it offers a practical testbed for evaluating coding agents that must modify C++ code within stateful, interactive, real-time systems. We present GameEngineBench, a benchmark for evaluating coding agents on scoped C++ implementation tasks inside Unreal Engine 5 projects, built from nine real-world game repositories. The evaluation set consists of 110 tasks spanning gameplay mechanics, multiplayer behavior, AI and world orchestration, animation and movement, UI and session code, loading behavior, online-service integration, persistence, data serialization, XR behavior, and rendering-oriented plugins. These tasks require models to make native C++ changes that compile and satisfy behavioral tests within executable Unreal Engine projects. Across twelve evaluated configurations, the strongest model reaches 55.5\% pass@1, while 31 tasks remain unsolved by every configuration. Our results demonstrate that frontier coding agents continue to struggle with deeply integrated C++ development for real-time interactive software, highlighting game-engine benchmarks as a valuable complement to existing software engineering evaluations.

Problem

• coding agents
• C++
• game engines
• real-time systems
• software evaluation

Innovation

• GameEngineBench
• coding agents
• C++ code generation
• Unreal Engine 5
• real-time interactive systems