SC2Arena and StarEvolve: Benchmark and Self-Improvement Framework for LLMs in Complex Decision-Making Tasks

2025-08-14

AI Summary

Existing LLM evaluation benchmarks for *StarCraft II* suffer from incomplete race coverage, coarse-grained action spaces, and weak observational representations, limiting their ability to assess realistic, complex decision-making. To address these limitations, we propose SC2Arena—the first open-source benchmark supporting all three races, native low-level action spaces, and optimized textual observations. We further introduce StarEvolve, a self-evolving framework integrating a planning-execution-verification闭环, high-quality sample scoring, and a hierarchical decision architecture enhanced by text-augmented spatial reasoning and iterative self-correction. Experiments demonstrate that StarEvolve significantly outperforms prior methods in strategic planning capability. SC2Arena establishes a new paradigm for evaluating and advancing generalist agents in complex, dynamic environments. All code, environments, and algorithms are publicly released.

Technology Category

• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Evolutionary Learning

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Evaluating large language models (LLMs) in complex decision-making is essential for advancing AI's ability for strategic planning and real-time adaptation. However, existing benchmarks for tasks like StarCraft II fail to capture the game's full complexity, such as its complete game context, diverse action spaces, and all playable races. To address this gap, we present SC2Arena, a benchmark that fully supports all playable races, low-level action spaces, and optimizes text-based observations to tackle spatial reasoning challenges. Complementing this, we introduce StarEvolve, a hierarchical framework that integrates strategic planning with tactical execution, featuring iterative self-correction and continuous improvement via fine-tuning on high-quality gameplay data. Its key components include a Planner-Executor-Verifier structure to break down gameplay, and a scoring system for selecting high-quality training samples. Comprehensive analysis using SC2Arena provides valuable insights into developing generalist agents that were not possible with previous benchmarks. Experimental results also demonstrate that our proposed StarEvolve achieves superior performance in strategic planning. Our code, environment, and algorithms are publicly available.

Problem

• Existing benchmarks lack full StarCraft II complexity coverage
• Need for improved LLM evaluation in strategic decision-making tasks
• Requirement for hierarchical self-improvement frameworks in gameplay

Innovation

• SC2Arena benchmark supports all StarCraft II complexities
• StarEvolve integrates strategic planning with tactical execution
• Planner-Executor-Verifier structure enables iterative self-correction