Exposing the Systematic Vulnerability of Open-Weight Models to Prefill Attacks

2026-02-16

AI Summary

This study addresses the vulnerability of open-weight large language models (LLMs) to prefill attacks during deployment, a threat inadequately mitigated by existing defenses. Through the first large-scale red-teaming evaluation, the work systematically assesses over twenty attack strategies—spanning both established and newly proposed methods—across multiple model families, incorporating customized adversarial prefix construction. The findings reveal that all major open-weight LLMs are highly susceptible to being manipulated into generating harmful content, with even models exhibiting strong reasoning capabilities failing to resist targeted attacks. By establishing prefill injection as a critical yet overlooked attack vector, this research provides essential empirical evidence and technical guidance for the development of robust defense mechanisms in future LLM deployments.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

As the capabilities of large language models continue to advance, so does their potential for misuse. While closed-source models typically rely on external defenses, open-weight models must primarily depend on internal safeguards to mitigate harmful behavior. Prior red-teaming research has largely focused on input-based jailbreaking and parameter-level manipulations. However, open-weight models also natively support prefilling, which allows an attacker to predefine initial response tokens before generation begins. Despite its potential, this attack vector has received little systematic attention. We present the largest empirical study to date of prefill attacks, evaluating over 20 existing and novel strategies across multiple model families and state-of-the-art open-weight models. Our results show that prefill attacks are consistently effective against all major contemporary open-weight models, revealing a critical and previously underexplored vulnerability with significant implications for deployment. While certain large reasoning models exhibit some robustness against generic prefilling, they remain vulnerable to tailored, model-specific strategies. Our findings underscore the urgent need for model developers to prioritize defenses against prefill attacks in open-weight LLMs.

Problem

• open-weight models
• prefill attacks
• systematic vulnerability
• large language models
• security

Innovation

• prefill attacks
• open-weight models
• systematic vulnerability
• red-teaming
• LLM security