SAINT: Attention-Based Modeling of Sub-Action Dependencies in Multi-Action Policies

2025-05-17

AI Summary

In real-world combinatorial action spaces, the exponential growth of sub-action combinations renders conventional reinforcement learning methods ineffective, while existing approaches struggle to capture complex joint dependencies among sub-actions. To address this, we propose a Transformer-based permutation-invariant policy network that models multi-component actions as unordered sets. Leveraging state-conditioned self-attention, our architecture end-to-end learns high-order interdependencies among sub-actions—bypassing restrictive factorization or sequential assumptions. The design inherently satisfies permutation invariance and scales to combinatorial action spaces with up to millions of joint actions. Evaluated on 15 diverse benchmark tasks spanning multiple domains, our method consistently outperforms strong baselines—including SAC and PPO—in both sample efficiency and policy performance.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Multiagent Systems: Adversarial Agents
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The combinatorial structure of many real-world action spaces leads to exponential growth in the number of possible actions, limiting the effectiveness of conventional reinforcement learning algorithms. Recent approaches for combinatorial action spaces impose factorized or sequential structures over sub-actions, failing to capture complex joint behavior. We introduce the Sub-Action Interaction Network using Transformers (SAINT), a novel policy architecture that represents multi-component actions as unordered sets and models their dependencies via self-attention conditioned on the global state. SAINT is permutation-invariant, sample-efficient, and compatible with standard policy optimization algorithms. In 15 distinct combinatorial environments across three task domains, including environments with nearly 17 million joint actions, SAINT consistently outperforms strong baselines.

Problem

• Exponential growth in action spaces limits RL algorithms
• Existing methods fail to capture complex joint behavior
• Modeling sub-action dependencies in multi-action policies

Innovation

• Transformer-based modeling of sub-action dependencies
• Permutation-invariant unordered set representation
• Self-attention conditioned on global state