Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning

2026-04-20

AI Summary

This work addresses the instability and overfitting commonly observed in off-policy reinforcement learning when employing large-scale critics. It introduces low-rank adaptation (LoRA) into critic learning for the first time, freezing a randomly initialized base matrix and optimizing only low-rank adapters to constrain updates within a low-dimensional subspace, thereby achieving structured sparse regularization. The proposed approach is compatible with the hyperspherical normalization geometry of SimbaV2 and integrates seamlessly into algorithms such as SAC and FastTD3. Evaluated on the DeepMind Control Suite and IsaacLab benchmarks, the method significantly reduces critic loss and enhances policy performance, demonstrating remarkable simplicity, scalability, and effectiveness.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Learning & Optimization for ROB
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Scaling critic capacity is a promising direction for enhancing off-policy reinforcement learning (RL). However, larger critics are prone to overfitting and unstable in replay-buffer-based bootstrap training. This paper leverages Low-Rank Adaptation (LoRA) as a structural-sparsity regularizer for off-policy critics. Our approach freezes randomly initialized base matrices and solely optimizes low-rank adapters, thereby constraining critic updates to a low-dimensional subspace. Built on top of SimbaV2, we further develop a LoRA formulation, compatible with SimbaV2, that preserves its hyperspherical normalization geometry under frozen-backbone training. We evaluate our method with SAC and FastTD3 on DeepMind Control locomotion and IsaacLab robotics benchmarks. LoRA consistently achieves lower critic loss during training and stronger policy performance. Extensive experiments demonstrate that adaptive low-rank updates provide a simple, scalable, and effective structural regularization for critic learning in off-policy RL.

Problem

• off-policy reinforcement learning
• critic learning
• overfitting
• training instability
• structural regularization

Innovation

• Low-Rank Adaptation
• Critic Regularization
• Off-Policy Reinforcement Learning
• Structural Sparsity
• SimbaV2