Defending Against Harmful Supervision Hidden in Benign Samples

2026-06-29

AI Summary

Current defense mechanisms struggle to detect covert harmful supervision signals embedded within seemingly benign instruction-tuning data, allowing models to be surreptitiously manipulated during fine-tuning. To address this vulnerability, this work introduces a novel threat model termed “Embedded Attack” and proposes Dual-Reference Supervised Fine-Tuning (DR-SFT), which, for the first time, integrates contrastive learning principles into supervised fine-tuning. DR-SFT combines a DPO-like pairwise preference objective with token-level regularization to enable fine-grained suppression of malicious signals. Experimental results demonstrate that DR-SFT substantially enhances model robustness against embedded harmful supervision, outperforming conventional defense strategies such as data filtering while preserving task performance.

Abstract

Existing defenses are effective when harmful content is explicitly mixed into downstream fine-tuning data, but crafted samples can instead hide harmful supervision inside benign tasks. We propose Embedded Attack, where harmful QA pairs are embedded within benign training samples, and show that representative guardrails often fail to detect them at the example level. To address this, we propose Dual-Reference SFT (DR-SFT), which adapts DPO-style contrastive objective design to SFT through token-level regularization, mitigating harmful fine-tuning beyond coarse data filtering.

Problem

• harmful supervision
• benign samples
• embedded attack
• guardrails
• fine-tuning

Innovation

• Embedded Attack
• Dual-Reference SFT
• token-level regularization
• harmful supervision
• contrastive objective