Hierarchical Reinforcement Learning in StarCraft Micromanagement with Influence Maps and Cluster-based Scripts

2026-06-29

AI Summary

This work addresses the challenges of low sample efficiency and opaque decision-making in StarCraft micromanagement, which arise from high-dimensional state-action spaces, sparse and delayed rewards, and the inability of conventional hierarchical methods to adaptively decompose tasks. To overcome these issues, the authors propose the HRL-IM/CBS framework, which encodes global battlefield states using influence map hashing and enables dynamic local unit coordination through clustering-based scripts. The approach employs a hierarchical multi-Q-table architecture that separates decision-making into high-level cluster-policy selection and low-level tactical execution. Evaluated across six asymmetric scenarios, the method matches the performance of deep reinforcement learning baselines while significantly improving sample efficiency and offering an interpretable decision process grounded in Q-table reasoning.

Abstract

Real-time strategy (RTS) games present significant AI challenges, characterized by expansive state-action spaces arising from multi-unit coordination in continuous battlefields, and sparse delayed rewards stemming from final win/lose signals. Existing approaches face a trade-off between managing the dimensionality explosion of joint actions and maintaining the interpretability of complex state representations. This complexity is further intensified by the limitation of traditional hierarchical structures in adaptively decomposing tasks into effective tactical modules. Such difficulties are compounded by the black-box nature of deep learning models and their reliance on sparse rewards, which together result in limited sample efficiency and a lack of decision-making transparency. To address these limitations, this paper proposes HRL-IM/CBS, a hierarchical reinforcement learning framework with influence map hashing and cluster-based scripts for StarCraft micromanagement. Influence map hashing encodes global battlefield situations into compact hexadecimal codes, capturing spatial control and relative advantage. Cluster-based scripts enable dynamic local coordination through adaptive unit partitioning. The hierarchical multi-Q-table architecture decomposes decision-making into upper-level clustering strategy selection and lower-level tactical execution, with reward allocation providing dense learning signals. Experiments across six asymmetric scenarios demonstrate competitive performance against deep RL baselines while offering advantages in sample efficiency and interpretability through transparent Q-table representations.

Problem

• Hierarchical Reinforcement Learning
• StarCraft Micromanagement
• Sparse Rewards
• Interpretability
• State-Action Space

Innovation

• Hierarchical Reinforcement Learning
• Influence Map Hashing
• Cluster-based Scripts
• Sample Efficiency
• Interpretability