Revisiting Action Factorization for Complex Action Spaces

2026-06-24

AI Summary

This study addresses the lack of systematic benchmarks in reinforcement learning for hybrid discrete-continuous action spaces by comprehensively evaluating multiple action factorization approaches—including independent networks, shared encoders, VDN, QPLEX, joint output, and autoregressive methods—across PPO, SAC, and DQN algorithms and three action space configurations. The evaluation spans 220 experimental setups across four lightweight environments. Key contributions include the introduction of two novel variants, VDN-PPO and PPO-MIX, featuring a branched critic architecture; the release of two new Gymnasium/PettingZoo-compliant environments, CoopPush and Hybrid-Shoot; and empirical findings showing that autoregressive methods generally achieve the best performance, native continuous SAC leads in effectiveness at higher computational cost, and the branched dual-stream architecture offers the most favorable trade-off between efficiency and performance.

Abstract

Many real-world control problems involve hybrid discrete-continuous action spaces. For example, steering and signaling in autonomous driving, and aiming and firing in robotics or video-games. Despite real-world hybrid factorization and reinforcement learning framework support for complex action spaces (e.g., Gymnasium, PettingZoo, TorchRL, SeedRL, Mujoco, etc), the default environments within those frameworks often implement uniform action space configurations (LunarLander, Walker2D, Cheetah, SMAC, SUMO, Ant, Atari). Landmark hybrid-action benchmarks (RoboCup 2D HFO, SC2LE, Platform, CARLA, etc) are mostly heavyweight or archival implementations originating from papers which test one or a small number of competing factorization methods on one kind of control. This article provides a cross-sectional study of factorization methods [independent networks, shared encoder, VDN, QPLEX, Joint, Auto-Regressive] on each of three families of algorithms [PPO, SAC, DQN] across three action spaces [discretized, hybrid, continuous] over four lightweight environments [Platform, hybrid-LunarLander, Hybrid-Shoot, CoopPush]. Accounting for some invalid pairings such as joint-continuous, we are left with 220 configurations to analyze each method. We provide two new C++ parallel gymnasium and petting-zoo compliant environments [CoopPush, Hybrid-Shoot] to isolate particular challenges such as state-dependent inter-action dependence. Finally, we introduce VDN-PPO and PPO-MIX which use a branching critic to assign credit to multi-headed PPO. These variants out-perform all other tested PPO factorizations. Our results suggest that branching dueling architectures balance compute and performance most effectively, with Auto-Regressive actions reaching the highest performance overall and native continuous SAC outperforming discrete and hybrid algorithms, albiet both at increased computational cost.

Problem

• hybrid action spaces
• action factorization
• reinforcement learning
• benchmarking
• discrete-continuous control

Innovation

• action factorization
• hybrid action spaces
• branching critic
• reinforcement learning
• cross-sectional study