HarDBench: A Benchmark for Draft-Based Co-Authoring Jailbreak Attacks for Safe Human-LLM Collaborative Writing

2026-04-21

AI Summary

This work identifies and addresses a critical vulnerability in large language models (LLMs) during collaborative writing: malicious users can exploit incomplete drafts to induce harmful outputs through draft injection jailbreak attacks. To systematically evaluate this risk, the authors introduce HarDBench, a structured benchmark encompassing high-risk domains such as explosives, narcotics, weapons, and cyberattacks. They further propose a safety-utility balanced alignment method based on preference optimization. Experimental results demonstrate that current LLMs are highly susceptible to such attacks, whereas the proposed approach effectively suppresses harmful content generation without compromising the model’s ability to collaborate efficiently on benign drafts.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models (LLMs) are increasingly used as co-authors in collaborative writing, where users begin with rough drafts and rely on LLMs to complete, revise, and refine their content. However, this capability poses a serious safety risk: malicious users could jailbreak the models-filling incomplete drafts with dangerous content-to force them into generating harmful outputs. In this paper, we identify the vulnerability of current LLMs to such draft-based co-authoring jailbreak attacks and introduce HarDBench, a systematic benchmark designed to evaluate the robustness of LLMs against this emerging threat. HarDBench spans a range of high-risk domains-including Explosives, Drugs, Weapons, and Cyberattacks-and features prompts with realistic structure and domain-specific cues to assess the model susceptibility to harmful completions. To mitigate this risk, we introduce a safety-utility balanced alignment approach based on preference optimization, training models to refuse harmful completions while remaining helpful on benign drafts. Experimental results show that existing LLMs are highly vulnerable in co-authoring contexts and our alignment method significantly reduces harmful outputs without degrading performance on co-authoring capabilities. This presents a new paradigm for evaluating and aligning LLMs in human-LLM collaborative writing settings. Our new benchmark and dataset are available on our project page at https://github.com/untae0122/HarDBench

Problem

• jailbreak attacks
• collaborative writing
• LLM safety
• harmful content generation
• draft-based co-authoring

Innovation

• draft-based jailbreak
• co-authoring safety
• HarDBench
• preference optimization
• human-LLM collaboration