Multilinear Tensor Low-Rank Approximation for Policy-Gradient Methods in Reinforcement Learning

2025-01-08

AI Summary

Policy gradient methods suffer from low parameter efficiency, sensitivity to architecture and hyperparameters, and difficulty in modeling state-action similarity. To address these issues, this paper proposes a policy modeling framework based on multilinear tensor low-rank approximation: policy parameters are structured as a high-order tensor and compactly represented via PARAFAC decomposition; crucially, tensor completion and low-rank constraints are introduced into policy parameterization for the first time. Theoretically, the method guarantees convergence across multiple policy classes while circumventing inherent limitations of neural networks. Empirical evaluation on standard benchmarks shows that the approach achieves cumulative reward comparable to full-parameter neural networks, while significantly reducing computational overhead and sample complexity. This yields substantial improvements in parameter efficiency and generalization robustness of policy learning.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Learning to Search
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Reinforcement learning (RL) aims to estimate the action to take given a (time-varying) state, with the goal of maximizing a cumulative reward function. Predominantly, there are two families of algorithms to solve RL problems: value-based and policy-based methods, with the latter designed to learn a probabilistic parametric policy from states to actions. Most contemporary approaches implement this policy using a neural network (NN). However, NNs usually face issues related to convergence, architectural suitability, hyper-parameter selection, and underutilization of the redundancies of the state-action representations (e.g. locally similar states). This paper postulates multi-linear mappings to efficiently estimate the parameters of the RL policy. More precisely, we leverage the PARAFAC decomposition to design tensor low-rank policies. The key idea involves collecting the policy parameters into a tensor and leveraging tensor-completion techniques to enforce low rank. We establish theoretical guarantees of the proposed methods for various policy classes and validate their efficacy through numerical experiments. Specifically, we demonstrate that tensor low-rank policy models reduce computational and sample complexities in comparison to NN models while achieving similar rewards.

Problem

• Reinforcement Learning
• Policy Gradient Methods
• Learning Efficiency

Innovation

• Multilinear Tensor
• PARAFAC Decomposition
• Low-Rank Tensor Policy