Sample-efficient Transfer Reinforcement Learning via Adaptive Reward Shaping and Policy-Ratio Reweighting Strategy

2026-06-24

AI Summary

This work addresses training instability, performance degradation, and high-risk exploration in transfer reinforcement learning caused by distributional shifts between source and target domains. To this end, the authors propose a safe and efficient transfer framework that integrates an adaptive teacher intervention mechanism to constrain hazardous actions, employs dynamically decaying reward shaping to enhance learning safety, and introduces a policy likelihood ratio–based sample weighting scheme to stabilize knowledge transfer. Theoretical analysis establishes a performance bound under a mixed behavior policy. Experimental results on diverse traffic densities and the real-world NGSIM dataset demonstrate that the proposed method not only ensures safety but also improves safety performance by over 52.2% and sample efficiency by 5.0%, significantly outperforming baseline approaches.

Abstract

Transfer learning improves policy learning efficiency by reusing knowledge from source tasks, providing a feasible paradigm for safe and efficient autonomous highway lane changing decision-making. Existing methods frequently encounter transfer mismatch induced by distribution shifts between source and target domains, leading to training oscillation and performance decline. Besides, target domain adaptation depends on exploratory interactions, which struggles to guarantee training safety in safety-critical lane changing cases. To tackle these limitations, this paper proposes a safe transfer reinforcement learning framework for autonomous highway lane changing. First, we design an adaptive teacher intervention mechanism based on instantaneous safety cost to restrain risky exploration and fade intervention strength progressively, with theoretical analysis on return bounds for mixed behavior policy. This intervention also produces dual-source samples for joint training. Second, a teacher-guided safe transfer module embeds action evaluation information of teacher policy into student learning via reward shaping to boost training safety and efficiency, with teacher guidance decaying as policy safety rises. Third, a teacher-guided weighted optimization mechanism adjusts sample weights in policy optimization using a likelihood ratio factor to stabilize transfer performance. Experiments under varied traffic densities and validations on real-world NGSIM dataset reveal that our method surpasses baseline approaches by over 52.2% in safety and 5.0% in learning efficiency. Results verify the efficacy and robustness of our safety-aware transfer strategy for autonomous highway lane changing under various traffic conditions.

Problem

• transfer mismatch
• distribution shift
• safe exploration
• autonomous lane changing
• reinforcement learning

Innovation

• adaptive reward shaping
• policy-ratio reweighting
• safe transfer reinforcement learning
• teacher-student intervention
• distribution shift mitigation