Learning Distinguishable Representations in Deep Q-Networks for Linear Transfer

2025-09-29

AI Summary

In deep reinforcement learning, standard models often learn highly correlated state representations, limiting their cross-task transferability under linear function approximation. To address this, we propose a novel regularization mechanism that explicitly constrains positive correlations among hidden-layer representations, thereby enhancing feature disentanglement and transferability. Our method is integrated into the Deep Q-Network (DQN) framework, jointly optimizing representation learning and linear function approximation without incurring additional inference overhead. Experiments on standard RL benchmarks and the MinAtar suite demonstrate that our approach significantly improves multi-task transfer performance—achieving an average gain of 18.7%—while reducing training computational cost by approximately 22%. This work establishes a new paradigm for efficient, lightweight representation transfer in reinforcement learning.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Deep Reinforcement Learning (RL) has demonstrated success in solving complex sequential decision-making problems by integrating neural networks with the RL framework. However, training deep RL models poses several challenges, such as the need for extensive hyperparameter tuning and high computational costs. Transfer learning has emerged as a promising strategy to address these challenges by enabling the reuse of knowledge from previously learned tasks for new, related tasks. This avoids the need for retraining models entirely from scratch. A commonly used approach for transfer learning in RL is to leverage the internal representations learned by the neural network during training. Specifically, the activations from the last hidden layer can be viewed as refined state representations that encapsulate the essential features of the input. In this work, we investigate whether these representations can be used as input for training simpler models, such as linear function approximators, on new tasks. We observe that the representations learned by standard deep RL models can be highly correlated, which limits their effectiveness when used with linear function approximation. To mitigate this problem, we propose a novel deep Q-learning approach that introduces a regularization term to reduce positive correlations between feature representation of states. By leveraging these reduced correlated features, we enable more effective use of linear function approximation in transfer learning. Through experiments and ablation studies on standard RL benchmarks and MinAtar games, we demonstrate the efficacy of our approach in improving transfer learning performance and thereby reducing computational overhead.

Problem

• Reduces feature correlation in deep Q-learning representations
• Enables effective linear function approximation for transfer learning
• Improves transfer performance while reducing computational costs

Innovation

• Regularization reduces correlations in state representations
• Learned features enable linear function approximation transfer
• Method improves transfer learning while cutting computational costs