Few-Shot Truly Benign DPO Attack for Jailbreaking LLMs

2026-05-09

AI Summary

This work demonstrates that large language models fine-tuned via preference optimization can covertly lose safety alignment even when exposed to an extremely small amount of harmless data. The authors propose a novel attack based on Direct Preference Optimization (DPO) that leverages only ten pairs of entirely benign preferences to significantly suppress the model’s refusal behavior toward harmful requests, without triggering detection by standard auditing mechanisms. Crucially, this attack generalizes to unseen prompts and is indistinguishable from legitimate user feedback aimed at reducing over-refusal, thereby achieving a “benign” jailbreak. The method reveals, for the first time, that DPO-based alignment is vulnerable to highly effective and stealthy safety failures under minimal-resource conditions. Evaluated on closed-source models such as GPT-4o, the attack achieves an 81.73% success rate at a cost of merely \$0.10; in open-source models, even a single preference pair suffices to induce the effect.

Abstract

Fine-tuning APIs make frontier LLMs easy to customize, but they can also weaken safety alignment during fine-tuning. While prior work shows that benign supervised fine-tuning (SFT) can reduce refusal behavior, deployed fine-tuning pipelines increasingly support preference-based objectives, whose safety risks remain less understood. We show that Direct Preference Optimization (DPO) introduces a stronger and harder-to-audit failure mode. We propose a truly benign DPO attack using only 10 harmless preference pairs, the minimum data scale accepted by OpenAI's fine-tuning service. Each pair contains a benign prompt, a normal helpful answer as the preferred response, and a refusal as the dispreferred response. Unlike prior benign fine-tuning attacks, our data exhibits no suspicious behavior: it is practically indistinguishable from the fine-tuning request of a legitimate user seeking to reduce over-refusal, making harmful intent almost impossible to infer from the request alone. Nevertheless, because DPO directly optimizes the model to prefer helpful answers over refusals, this seemingly benign objective broadly suppresses refusal behavior and transfers to harmful prompts outside the fine-tuning data. Across OpenAI models supporting DPO fine-tuning, our attack achieves attack success rates of 59.13% on GPT-4o, 70.20% on GPT-4.1, 54.80% on GPT-4.1-mini, and 81.73% on GPT-4.1-nano, at costs of only \$1.7, \$1.7, \$0.3, and \$0.1. Moreover, on open-weight models that do not impose minimum data requirements, we find that this effect can emerge from even a single benign preference pair.

Problem

• Few-Shot
• Benign DPO Attack
• Jailbreaking LLMs
• Safety Alignment
• Preference Optimization

Innovation

• Few-Shot DPO Attack
• Benign Preference Pairs
• Safety Alignment Erosion
• Jailbreaking LLMs
• Preference Optimization