Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMs

2026-07-09

AI Summary

This work addresses the challenge of dynamically selecting tasks of appropriate difficulty to construct open-ended curricula in reinforcement learning based on agent capabilities. It introduces, for the first time, a video language model into multi-agent reinforcement learning to enable fine-grained understanding of agent behaviors through direct analysis of policy execution videos—leveraging models such as VideoLLaMA2-7B—and to automatically generate adaptive curricula. Departing from conventional approaches that rely on scalar scores or textual summaries, the proposed method employs a vision-driven task recommendation mechanism. Evaluated on the StarCraft Multi-Agent Challenge (SMAC) benchmark, it significantly outperforms existing baselines, demonstrating both its effectiveness and scalability.

Abstract

Open-ended curricula in Reinforcement Learning (RL) aim to train generally-capable agents by identifying tasks that facilitate learning increasingly complex skills. A major challenge when designing such curricula is assessing task difficulty relative to the agent's current learning progress. While previous work has explored using scalar task scores or textual summaries of the agent's behavior, here we study a different approach: directly inspecting policy behavior via recorded episode videos. We introduce a simple yet effective instantiation of this approach which leverages a Video Language Model (VLM) to both process these videos and provide curriculum recommendations, which we call Visual Inspection of Policies (VIP). Since videos can naturally contain any number of controllable agents, we empirically study VIP on the StarCraft Multi-Agent Challenge (SMAC). We show that even with a lightweight and openly accessible VLM (VideoLLaMa2-7B), VIP can use policy videos to generate more effective curricula than both its text-only ablation and methods that rely on scalar task scores.

Problem

• open-ended curricula
• task difficulty assessment
• multi-agent reinforcement learning
• policy evaluation
• curriculum learning

Innovation

• Visual Inspection of Policies
• Video Language Model
• Open-ended Curriculum
• Multi-agent Reinforcement Learning
• Autocurricula