Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning

2026-07-08

AI Summary

This work addresses the challenge of effectively integrating human demonstrations and evaluative feedback to improve behavioral alignment of imitation learning agents in offline, full-sequence decision-making settings. The authors propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic approach that, for the first time, deeply fuses human demonstrations and evaluative feedback within a single-stage offline training framework, establishing a novel alignment learning paradigm tailored for sequential decision tasks. The method is evaluated on Safety Gymnasium, a benchmark designed for safety-aware reinforcement learning. Experimental results demonstrate that FMR consistently enhances alignment performance across multiple imitation learning algorithms, reducing misaligned behaviors by up to 98%, while maintaining robustness under data scarcity and noisy demonstrations.

Abstract

Reinforcement learning (RL) research has increasingly shifted focus towards alignment, ensuring agents learn behaviors adhering to human values. While human demonstrations and feedback have proven crucial for alignment, existing approaches predominantly combine these signals using multi-stage pipelines designed for the contextual bandit framing of language generation. Yet little work explores how these complementary inputs can serve as a richer, interconnected signal for single-stage offline training in fully sequential decision-making environments. We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies. We adapt Safety Gymnasium environments to be a principled testbed for alignment evaluation, demonstrating improved aptitude and up to a 98\% reduction in misalignment across a range of imitation learning algorithms. FMR remains robust in limited data regimes, even when learning from scarce aligned and uninformative noisy demonstrations.

Problem

• offline imitation learning
• agent alignment
• evaluative feedback
• sequential decision-making
• human demonstrations

Innovation

• Feedback Manipulation Regularization
• Offline Imitation Learning
• Agent Alignment
• Evaluative Feedback
• Sequential Decision-Making