FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions

2025-04-14

AI Summary

To address catastrophic reward forgetting (CRF)—a critical issue in robot behavior style adaptation where fine-tuning reward models on sparse preference data severely degrades original task performance—this paper proposes Low-Rank Reward Adaptation (LORA). LORA enables parameter-efficient fine-tuning of reward functions via low-rank matrix decomposition, jointly preserving both new style preferences and original task capabilities within a preference-based reinforcement learning (PbRL) framework. It is the first method to systematically mitigate CRF in PbRL while balancing preference alignment accuracy and task robustness. Experiments across multiple simulated and real-robot tasks demonstrate that LORA achieves precise style transfer using ≤50 preference pairs, incurs <3% performance degradation on original tasks, and improves sample efficiency by 5.2× compared to baseline approaches.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Preference-based reinforcement learning (PbRL) is a suitable approach for style adaptation of pre-trained robotic behavior: adapting the robot's policy to follow human user preferences while still being able to perform the original task. However, collecting preferences for the adaptation process in robotics is often challenging and time-consuming. In this work we explore the adaptation of pre-trained robots in the low-preference-data regime. We show that, in this regime, recent adaptation approaches suffer from catastrophic reward forgetting (CRF), where the updated reward model overfits to the new preferences, leading the agent to become unable to perform the original task. To mitigate CRF, we propose to enhance the original reward model with a small number of parameters (low-rank matrices) responsible for modeling the preference adaptation. Our evaluation shows that our method can efficiently and effectively adjust robotic behavior to human preferences across simulation benchmark tasks and multiple real-world robotic tasks.

Problem

• Adapting pre-trained robots to human preferences efficiently
• Preventing catastrophic reward forgetting in low-preference-data regimes
• Enhancing reward models with low-rank matrices for style adaptation

Innovation

• Low-rank matrices for reward adaptation
• Mitigates catastrophic reward forgetting
• Efficient style adaptation with few preferences