🤖 AI Summary
This work addresses performance bottlenecks in multi-agent reinforcement learning (MARL) arising from sparse rewards, high-dimensional state-action spaces, and the challenge of coordinated policy learning. The authors propose a hierarchical architecture wherein a pretrained large language model (LLM) serves as a centralized strategic controller at the high level, dynamically selecting among specialized low-level reinforcement learning policies without relying on handcrafted rules. This approach represents the first integration of LLMs into high-level planning for multi-agent systems, significantly enhancing tactical diversity and behavioral human-likeness. Evaluated on a 2v2 capture-the-flag task, the method achieves a win rate of 46.4%, matching the performance of hand-designed behavior trees and substantially outperforming flat RL baselines. A user study further reveals that 60% of participants judged the agents’ behavior as most human-like (p = 0.027).
📝 Abstract
Reinforcement learning (RL) has achieved strong performance in sequential decision-making, yet scaling to complex multi-agent environments remains challenging due to sparse rewards, large state-action spaces, and the difficulty of learning coordinated strategies. We propose a hierarchical architecture where a pretrained large language model (LLM) acts as a centralized strategic controller that selects among specialized RL skill policies for a team of agents, while RL policies handle reactive low-level execution. We evaluate this hybrid system in a competitive 2v2 King of the Hill environment against behavior tree (BT) and \emph{``Flat''} RL (end-to-end training without skill decomposition) baselines. The LLM+RL system achieves task performance statistically equivalent to hand-crafted BT (46.4\% vs 51.5\% win rate, $p=0.103$) while both significantly outperform Flat RL trained without skill decomposition. A user study ($n=15$) reveals that 60\% of participants perceive LLM+RL agents as the most human-like ($p=0.027$), citing behavioral adaptability and tactical variability. These results demonstrate that pretrained LLM reasoning can effectively orchestrate pretrained RL skills, achieving competitive multi-agent coordination and superior perceived believability without manual rule engineering.