PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment

2026-05-20

AI Summary

This work proposes a method to endow pretrained reinforcement learning policies with safety awareness and cost constraint satisfaction without requiring retraining from scratch. By leveraging trajectory-level preference data, it extends Direct Preference Optimization (DPO) to sequential decision-making in continuous control for the first time. The approach constructs counterfactual trajectories sampled from the current policy, enabling alignment with safety and cost constraints while preserving high-reward behaviors. Integrating implicit preference modeling, counterfactual trajectory generation, and DPO, the method substantially improves data and computational efficiency. Empirical results demonstrate that, while maintaining the original reward performance, it reduces constraint violations and catastrophic failures by over 60%.

Abstract

We address the problem of making a pre-trained reinforcement learning (RL) policy safety-aware by incorporating cost constraints without retraining it from scratch. While costs could be numerically encoded, we assume a more general setting is when costs are provided as preferences. Given a reward-optimized policy and a small dataset of preferred (low-cost) and dispreferred (high-cost) trajectories, our goal is to fine-tune the policy to generate low-cost behaviors while retaining high rewards. Unlike standard RLHF in language models, where preferences are defined over responses to the same prompt, our setting involves trajectory-level preferences in continuous control environments. We introduce PREFINE: Preference-based Implicit Reward and Cost Fine-Tuning for Safety Alignment which is a preference-based fine-tuning method that adapts Direct Preference Optimization (DPO), which is now widely used for LLM fine-tuning, to the sequential decision making setting. PREFINE constructs policy-sampled counterfactual trajectories to establish meaningful preference contrasts and jointly optimizes for reward retention and safety alignment. Empirically, PREFINE reduces constraint violations and catastrophic failures by over 60% while maintaining original reward behavior. PREFINE produces policies that achieve low-cost, high-reward performance with significantly improved data and computational efficiency compared to full offline RL or imitation learning, bridging preference alignment and safe policy adaptation in continuous domains.

Problem

• safety alignment
• preference-based learning
• cost constraints
• reinforcement learning
• trajectory preferences

Innovation

• Preference-based fine-tuning
• Safety alignment
• Direct Preference Optimization (DPO)
• Trajectory-level preferences
• Implicit reward and cost