PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMs

2026-05-21

AI Summary

This work addresses the vulnerability of instruction-tuned large language models to task-level backdoor poisoning attacks when trained on unverified data, proposing PoisonForge—a systematic benchmark for evaluating such threats. By injecting only a minimal number (e.g., 10 samples, or 1% of the training set) of carefully crafted instruction-response pairs, the attack reliably induces the model to output attacker-specified content on a targeted task while preserving near-original performance on others (leakage rate < 0.5%). The study introduces the first four-dimensional parameterization of task-level poisoning threats, demonstrating that attack success hinges primarily on poison design rather than model scale, and develops a generalizable risk prediction model. Evaluated across 12 mainstream open-source models, the approach achieves over 70% attack success rates in 11 models under the most vulnerable configurations.

Abstract

When practitioners fine-tune LLMs on unvetted datasets, an adversary can exploit the data supply chain through task-level poisoning: inserting a small number of crafted instruction-response pairs that cause the model to embed attacker-specified entities, such as a country, in outputs for a targeted task family while behaving normally elsewhere. We introduce PoisonForge, a benchmark that parameterizes this threat along four dimensions (bias type, poisoning mode, appearance count, and target output length) and evaluates 12 open-weight models (from 2B to 32B parameters) across five families under a primarily 1% poison budget. With only 10 poisoned examples among 1,000 fine-tuning examples, 11 of 12 models exceed a 70% attack success rate (ASR) in their most vulnerable configuration. Meanwhile, unintended leakage to non-target tasks remains below 0.5%, and models perform well on standard benchmarks. We analyze in detail the factors contributing to attack success. We observe that multiple appearances of an entity increase the ASR, the optimal poisoning mode depends on the semantic structure of the target entity, and ASR drops monotonically with the task output length. A correlation analysis and risk prediction model confirm that poisoning design choices, rather than model scale, are the primary causes of attack success, and that these patterns generalize to predict attack success on new tasks. We release all configurations, pipelines, and analysis code to support reproducible comparisons.

Problem

• task-level poisoning
• instruction-tuned LLMs
• data poisoning
• adversarial attack
• poisoning benchmark

Innovation

• task-level poisoning
• instruction-tuned LLMs
• PoisonForge benchmark
• targeted backdoor attack
• data supply chain security