Sim2O: Efficient Offline-to-Online MARL via Joint Action Composition

2026-06-19

AI Summary

This work addresses the challenges of inefficient offline data utilization and difficult coordinated exploration in offline-to-online transfer for multi-agent reinforcement learning. It proposes Sim2O, a minimalist framework that, for the first time, formulates this transfer process as a combinatorial problem over joint actions. Sim2O dynamically fuses offline and online action proposals from individual agents and leverages a centralized value function to evaluate hybrid action combinations, thereby enabling efficient discovery of coordinated policies. Notably, the method incurs no additional training objectives or architectural overhead. Extensive experiments across multiple benchmark tasks demonstrate that Sim2O significantly outperforms existing approaches, validating its effectiveness and efficiency in multi-agent offline-to-online adaptation.

Abstract

Offline-to-online adaptation serves as a pivotal paradigm for mitigating the prohibitive cost of online exploration by bootstrapping reinforcement learning from offline datasets. While this paradigm has been extensively studied in single-agent settings, its extension to Multi-Agent Reinforcement Learning (MARL) remains largely unexplored, despite its critical relevance to complex coordinated decision-making. To bridge this gap, we introduce Sim2O, an elegant and minimalist framework for offline-to-online MARL. Rather than treating adaptation as a monolithic joint decision, Sim2O conceptualizes it as a compositional process. Specifically, candidate joint actions are synthesized by dynamically blending offline and online action proposals across agents. By leveraging a centralized value function to evaluate these hybrid combinations, Sim2O identifies high-value coordination strategies without requiring auxiliary training objectives or structural overhead. Empirical evaluations across diverse benchmarks demonstrate that Sim2O significantly outperforms existing baselines, underscoring that a minimalist design is not only viable but highly effective for multi-agent offline-to-online adaptation.

Problem

• offline-to-online adaptation
• Multi-Agent Reinforcement Learning
• coordinated decision-making
• online exploration
• offline datasets

Innovation

• offline-to-online adaptation
• multi-agent reinforcement learning
• joint action composition
• centralized value function
• minimalist framework