Scouting By Reward: VLM-TO-IRL-Driven Player Selection For Esports

2026-04-15

AI Summary

This work proposes a multimodal inverse reinforcement learning framework to address the limitations of traditional esports player selection, which relies on manual replay analysis and aggregated performance metrics and struggles to identify decision-making patterns aligned with specific tactical styles. The proposed approach uniquely integrates in-game telemetry data with tactical commentaries generated by vision-language models. Through a dual-branch architecture, it jointly models structured state-action trajectories and semantic tactical descriptions, learning player-specific reward functions via a Generative Adversarial Imitation Learning (GAIL) objective. This enables precise talent identification based on tactical compatibility rather than general skill alone. The method is embedded within a scalable digital twin system, facilitating data-driven team composition and targeted talent discovery from large candidate pools.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Teamwork, Team formation
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Traditional esports scouting workflows rely heavily on manual video review and aggregate performance metrics, which often fail to capture the nuanced decision-making patterns necessary to determine if a prospect fits a specific tactical archetype. To address this, we reframe style-based player evaluation in esports as an Inverse Reinforcement Learning (IRL) problem. In this paper, we introduce a novel player selection framework that learns professional-specific reward functions from logged gameplay demonstrations, allowing organizations to rank candidates by their stylistic alignment with a target star player. Our proposed architecture utilizes a multimodal, two-branch intake: one branch encodes structured state-action trajectories derived from high-resolution in-game telemetry, while the second encodes temporally aligned tactical pseudo-commentary generated by Vision-Language Models (VLMs) from broadcast footage. These representations are fused and evaluated via a Generative Adversarial Imitation Learning (GAIL) objective, where a discriminator learns to capture the unique mechanical and tactical signatures of elite professionals. By transitioning from generic skill estimation to scouting "by reward," this framework provides a scalable, workflow-aware digital twin system that enables data-driven roster construction and targeted talent discovery across massive candidate pools.

Problem

• esports scouting
• player selection
• tactical archetype
• decision-making patterns
• style-based evaluation

Innovation

• Inverse Reinforcement Learning
• Vision-Language Models
• Generative Adversarial Imitation Learning
• Player Scouting
• Multimodal Fusion