HyPOLE: Hyperproperty-Guided Multi-Agent Reinforcement Learning under Partial Observation

2026-06-29

AI Summary

This work addresses the lack of rigorous formal mechanisms for specifying collaborative objectives and safety constraints in partially observable multi-agent reinforcement learning. To this end, the paper introduces HyPOLE, a novel framework that, for the first time, incorporates HyperLTL—a hyperproperty temporal logic—into this domain to precisely specify global temporal properties over multi-agent systems. HyPOLE integrates these formal specifications within the centralized training with decentralized execution (CTDE) paradigm to guide the learning of decentralized policies. Experimental results on benchmark environments including SMAC, MessySMAC, and WildFire demonstrate that HyPOLE significantly outperforms existing approaches, thereby validating the efficacy and superiority of formal specification–guided learning in enhancing both policy performance and safety.

Abstract

Formal specification is a powerful tool to guide the learning process and provides significant advantages over reward shaping: (1) mathematical rigor; (2) expressiveness to specify objectives and constraints, and (3) the ability to define tactics to achieve objectives. However, these benefits remain largely unexplored in the context of Multi-Agent Reinforcement Learning (MARL). This paper introduces HyPOLE, a novel framework for MARL under partial observability, where learning is guided by the expressive power of the so-called hyperproperties and, in particular, the temporal logic HyperLTL. We integrate Centralized Training for Decentralized Execution (CTDE) techniques with HyPOLE to synthesize decentralized policies, and our evaluation on SMAC, MessySMAC, and WildFire benchmark demonstrates clear advantages over baselines.

Problem

• Multi-Agent Reinforcement Learning
• Partial Observability
• Hyperproperties
• Formal Specification
• HyperLTL

Innovation

• Hyperproperties
• HyperLTL
• Multi-Agent Reinforcement Learning
• Partial Observability
• Formal Specification