Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces

2026-07-01

AI Summary

This work addresses a structural security vulnerability in large language models (LLMs) operating in function-calling scenarios, where ambiguous trust boundaries render conventional prompt-based defenses ineffective against multi-turn jailbreaking attacks. The authors propose SMT (Simulated Moderation Traces), a black-box attack framework that progressively weakens model safety constraints by crafting multi-round dialogues masquerading as legitimate moderation processes. SMT leverages structured tool-call contexts, forged moderation frames, and feedback-driven iterative optimization to bypass safeguards across conversation turns. This approach uncovers a previously unexplored attack surface in function-calling LLMs, transcending the limitations of single-turn prompt injection paradigms. Evaluated on five leading commercial models, SMT achieves the highest average attack success rate and HarmScore with the fewest queries, underscoring the critical need for context-aware security validation mechanisms.

Abstract

Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs). While prior work has primarily studied attacks and defenses at the prompt level, we show that this prompt-centric paradigm overlooks a structural vulnerability in stateful, function-calling environments. In such applications, developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context. This architecture expands the attack surface by blurring the boundary between trusted control logic and untrusted data, allowing adversarial intent to be distributed across a multi-turn execution path. We exploit this architectural flaw through SMT, a black-box attack framework based on Simulated Moderation Traces. Departing from purely prompt-based interactions, SMT constructs a multi-turn trajectory that simulates a legitimate moderation-auditing workflow. Within this trajectory, a fabricated moderation frame leverages red-team testing as a pretext to elicit harmful generations. The subsequent validation feedback treats safety refusals as execution failures, prompting refinements that gradually weaken the model's safety constraints and ultimately trigger harmful outputs. Extensive empirical evaluations on prominent commercial LLMs from five different providers across two standardized safety benchmarks show that SMT consistently achieves the highest average attack success rate and HarmScore while requiring a near-minimal number of queries, substantially outperforming existing baselines. These findings demonstrate that prompt-level sanitization alone is fundamentally insufficient for defending tool-enabled LLM systems and highlight the urgent need for context-aware validation across schemas, arguments, tool outputs, and accumulated conversation state. The code is available at https://github.com/liujlong27/SMT.

Problem

• jailbreak attacks
• function-calling LLMs
• structural vulnerability
• multi-turn execution
• safety bypass

Innovation

• function-calling LLMs
• jailbreak attack
• Simulated Moderation Traces
• context-aware validation
• multi-turn execution