Vector Symbolic Policy Gradient

2026-08-18

AI Summary

本文提出Vector-Symbolic Policy Gradient方法，通过将每个动作表示为超向量并根据状态编码相似性评分来解决离散动作策略梯度问题。

Abstract

We answer this question with Vector-Symbolic Policy Gradient (VSPG), a discrete-action actor that represents each action by a unit-norm hypervector and scores it by similarity to the encoded state. Under the standard softmax policy-gradient surrogate, we prove that its update is exactly advantage-weighted hypervector bundling followed by normalization, and therefore supports standard advantage estimators. We further show that each trained action hypervector is a fixed-size compressed kernel memory, storing an advantage-weighted kernel expansion over visited states and transferring evidence according to the encoder-induced similarity. This provides a concrete mechanism that can support sample-efficient learning without increasing inference-time memory. Finally, for bipolar action memories, we prove that greedy action selection is stable under random bit flips, with failure probability decaying exponentially in the hypervector dimension. VSPG thus connects VSA action memories, log-linear policy gradients, and kernel policy search while providing a quantitative robustness guarantee.

Problem

• Vector-Symbolic Policy Gradient
• discrete-action actor
• hypervector
• advantage estimators
• robustness

Innovation

• Vector-Symbolic Policy Gradient
• advantage-weighted hypervector bundling
• compressed kernel memory
• robustness guarantee