Is Per-Agent Policy Composition Safe? Rethinking Successor-Feature Transfer in Cooperative Multi-Agent Reinforcement Learning

2026-08-12

AI Summary

In cooperative multi-agent reinforcement learning, independently learned policies may yield unsafe joint behaviors when deployed under dynamic goals. To address this challenge, this work proposes MA-USFA, the first synchronization-conditioned composition mechanism that simultaneously ensures safety and goal flexibility. The approach employs a hierarchical architecture: a lower-level conditional universal successor feature approximator and an upper-level cross-agent policy selection and correction module, integrated with generalized policy improvement to enable safe composition. Notably, MA-USFA requires no task-specific fine-tuning and can be safely deployed on any new task within the target distribution, significantly outperforming baseline methods based on independent policy composition.

Abstract

Many reinforcement learning systems, from fleet management to traffic signal control, must serve an objective that changes dynamically after deployment, and retraining a policy for each new objective is prohibitively expensive. For a single agent, this problem is well understood: successor features with generalized policy improvement, together with their universal extension, recombine a library of learned policies into a policy for any new objective, with a guarantee that the result is never worse than any policy in the library. However, multi-agent transfer has received far less attention, and the common practice of letting each agent recombine its own library independently inherits the recipe but not the guarantee. We prove that this independent composition can produce joint behavior strictly worse than every policy in the library, because recombining teammates changes the environment each agent faces and invalidates the values it relies on, a failure with no single-agent counterpart. We further show that the only unconditionally safe fixed rule is synchronized composition, which moves the whole team to one jointly trained policy but cannot serve objectives that assign different goals to different agents. To attain safety and flexibility at once, we propose MA-USFA, a hierarchical method with two layers: a lower layer of universal successor feature approximators that predicts each agent's successor features while conditioned on its teammates' objectives, and an upper composer that selects, across agents, which library entry each agent should follow and supplies the cross-agent correction a per-agent value cannot represent. Trained once over the distribution of objectives, it is applied at deployment with no per-task adaptation.

Problem

• multi-agent reinforcement learning
• policy composition
• successor features
• transfer learning
• safety guarantee

Innovation

• multi-agent reinforcement learning
• successor features
• policy composition
• transfer safety
• universal successor feature approximators