TEMPLATEFUZZ: Fine-Grained Chat Template Fuzzing for Jailbreaking and Red Teaming LLMs

2026-04-13

AI Summary

This work addresses the vulnerability of large language models (LLMs) to jailbreaking attacks, noting that prior research has predominantly focused on prompt injection while overlooking chat templates as a critical attack surface. To bridge this gap, the authors introduce TEMPLATEFUZZ, a novel framework that applies fuzzing at the chat template level through element-wise mutation, heuristic search, and active learning to jointly optimize attack success rate (ASR) and model accuracy. The approach features a lightweight, rule-based jailbreak evaluator to systematically identify high-risk templates. Experimental results demonstrate an average ASR of 98.2% across twelve open-source LLMs with only a 1.1% drop in accuracy, and achieve a 90% average ASR on five commercial LLMs via template injection, substantially outperforming existing methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) are increasingly deployed across diverse domains, yet their vulnerability to jailbreak attacks, where adversarial inputs bypass safety mechanisms to elicit harmful outputs, poses significant security risks. While prior work has primarily focused on prompt injection attacks, these approaches often require resource-intensive prompt engineering and overlook other critical components, such as chat templates. This paper introduces TEMPLATEFUZZ, a fine-grained fuzzing framework that systematically exposes vulnerabilities in chat templates, a critical yet underexplored attack surface in LLMs. Specifically, TEMPLATEFUZZ (1) designs a series of element-level mutation rules to generate diverse chat template variants, (2) proposes a heuristic search strategy to guide the chat template generation toward the direction of amplifying the attack success rate (ASR) while preserving model accuracy, and (3) integrates an active learning-based strategy to derive a lightweight rule-based oracle for accurate and efficient jailbreak evaluation. Evaluated on twelve open-source LLMs across multiple attack scenarios, TEMPLATEFUZZ achieves an average ASR of 98.2% with only 1.1% accuracy degradation, outperforming state-of-the-art methods by 9.1%-47.9% in ASR and 8.4% in accuracy degradation. Moreover, even on five industry-leading commercial LLMs where chat templates cannot be specified, TEMPLATEFUZZ attains a 90% average ASR via chat template-based prompt injection attacks.

Problem

• jailbreak attacks
• chat templates
• Large Language Models
• red teaming
• adversarial inputs

Innovation

• chat template fuzzing
• jailbreaking
• red teaming
• element-level mutation
• active learning