SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Control

2026-05-01

AI Summary

This work proposes SAVGO, a novel reinforcement learning algorithm that explicitly incorporates the geometric structure of value functions into policy optimization—a perspective largely overlooked by existing methods. By constructing a joint state-action embedding space where pairs with similar values exhibit high cosine similarity, SAVGO leverages this geometry to design a similarity kernel that guides policy updates toward high-value regions. The approach unifies representation learning, value estimation, and policy optimization under a coherent geometric objective. Evaluated on high-dimensional continuous control tasks in MuJoCo, SAVGO significantly outperforms strong baselines, and ablation studies confirm the effectiveness of both the value-geometry modeling and the similarity-guided policy update mechanism.

Abstract

While representation and similarity learning have improved the sample efficiency of Reinforcement Learning (RL), they are rarely used to shape policy updates directly in the action space. To bridge this gap, a geometry-aware RL algorithm that explicitly incorporates value-based similarity into the policy update, State-Action Value Geometry Optimization (SAVGO), is proposed. In detail, SAVGO learns a joint state-action embedding space in which pairs with similar action-value estimates exhibit high cosine similarity, while dissimilar pairs are mapped to distinct directions. This learned geometry enables the generation of a similarity kernel over candidate actions sampled at each update, allowing policy improvement to be guided directly toward higher-value regions beyond local gradient-based updates. As a result, representation learning, value estimation, and policy optimization are unified within a single geometry-consistent objective, while preserving the scalability of off-policy actor-critic training. The proposed method is evaluated on standard MuJoCo continuous-control benchmarks, demonstrating improvements over strong baselines on challenging high-dimensional tasks. Ablation studies are done to analyze the contributions of value-geometry learning and similarity-based policy updates.

Problem

• reinforcement learning
• policy update
• action space
• similarity learning
• continuous control

Innovation

• value geometry
• cosine similarity
• policy optimization
• representation learning
• continuous control