Multi-agent imitation learning with function approximation: Linear Markov games and beyond

2026-02-26

AI Summary

This work addresses the challenge of multi-agent imitation learning in Markov games, where performance often degrades due to state-action distribution shift under overly restrictive concentrability assumptions. Focusing on linear Markov games, the authors introduce a novel concentrability condition defined in feature space, substantially relaxing the requirement on expert coverage. Building upon this, they propose the first computationally efficient, interactive multi-agent imitation learning algorithm that operates without any concentrability coefficient. Theoretical analysis shows that the sample complexity depends only on the feature dimension \(d\), independent of the state-action space size. Empirical evaluations on benchmark games such as Tic-Tac-Toe and Connect4 demonstrate that the proposed deep interactive algorithm significantly outperforms behavioral cloning.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

In this work, we present the first theoretical analysis of multi-agent imitation learning (MAIL) in linear Markov games where both the transition dynamics and each agent's reward function are linear in some given features. We demonstrate that by leveraging this structure, it is possible to replace the state-action level "all policy deviation concentrability coefficient" (Freihaut et al., arXiv:2510.09325) with a concentrability coefficient defined at the feature level which can be much smaller than the state-action analog when the features are informative about states' similarity. Furthermore, to circumvent the need for any concentrability coefficient, we turn to the interactive setting. We provide the first, computationally efficient, interactive MAIL algorithm for linear Markov games and show that its sample complexity depends only on the dimension of the feature map $d$. Building on these theoretical findings, we propose a deep MAIL interactive algorithm which clearly outperforms BC on games such as Tic-Tac-Toe and Connect4.

Problem

• multi-agent imitation learning
• linear Markov games
• concentrability coefficient
• sample complexity
• interactive setting

Innovation

• multi-agent imitation learning
• linear Markov games
• feature-level concentrability
• interactive learning
• sample complexity