Efficient Preference Poisoning Attack on Offline RLHF

2026-05-04

AI Summary

This work addresses the vulnerability of offline reinforcement learning from human feedback (RLHF) systems to label-flipping poisoning attacks when using pre-collected preference data. Focusing on the log-linear Direct Preference Optimization (DPO) algorithm, the study formulates poisoning for the first time as a structured binary sparse approximation problem, revealing that label flips induce a gradient bias independent of model parameters. Leveraging this insight, the authors develop the first minimally invasive poisoning framework with theoretical guarantees. They propose two efficient attack methods—Binary Lattice Basis Reduction Attack (BAL-A) and Binary Matching Pursuit Attack (BMP-A)—which integrate the LLL algorithm and Babai’s nearest plane method to handle non-normalized gradient dictionaries. Experiments on synthetic dictionaries and the Stanford Human Preferences dataset demonstrate the attacks’ efficacy, showing that dictionary geometry governs success rates and can yield certificates of robustness infeasibility.

Abstract

Offline Reinforcement Learning from Human Feedback (RLHF) pipelines such as Direct Preference Optimization (DPO) train on a pre-collected preference dataset, which makes them vulnerable to preference poisoning attack. We study label flip attacks against log-linear DPO. We first illustrate that flipping one preference label induces a parameter-independent shift in the DPO gradient. Using this key property, we can then convert the targeted poisoning problem into a structured binary sparse approximation problem. To solve this problem, we develop two attack methods: Binary-Aware Lattice Attack (BAL-A) and Binary Matching Pursuit Attack (BMP-A). BAL-A embeds the binary flip selection problem into a binary-aware lattice and applies Lenstra-Lenstra-Lovász reduction and Babai's nearest plane algorithm; we provide sufficient conditions that enforce binary coefficients and recover the minimum-flip objective. BMP-A adapts binary matching pursuit to our non-normalized gradient dictionary and yields coherence-based recovery guarantees and robustness (impossibility) certificates for $K$-flip budgets. Experiments on synthetic dictionaries and the Stanford Human Preferences dataset validate the theory and highlight how dictionary geometry governs attack success.

Problem

• Preference Poisoning Attack
• Offline RLHF
• Direct Preference Optimization
• Label Flip Attack
• Adversarial Robustness

Innovation

• Preference Poisoning
• Direct Preference Optimization
• Binary Sparse Approximation
• Lattice Reduction
• Matching Pursuit