Decoding-Level Taboo: A Diagnostic Stress Test for LLM Robustness

2026-08-10

AI Summary

This work addresses the performance degradation of large language models (LLMs) in real-world deployment, where system prompts or safety constraints often divert generation from standard pathways despite strong benchmark results. To diagnose such robustness gaps, the authors propose a prompt-free stress-testing methodology that dynamically intervenes in the logit space during decoding by masking dominant candidate tokens at word boundaries, thereby compelling the model to generate via alternative, non-canonical paths. This approach introduces a novel runtime dynamic token-masking mechanism, establishing a zero-prompt stress-testing paradigm. Experiments across multiple open-source LLMs reveal significant effects of model scale and instruction alignment on robustness under such perturbations. Furthermore, the method efficiently produces diverse synthetic data, enabling pre-deployment reliability auditing and evaluation of safety mechanisms.

Abstract

Large language model evaluations typically focus on performance under nominal conditions, creating an illusion of capability where models comfortably walk a narrow, highly optimized generation corridor. In real-world deployments, however, complex system prompts, safety guardrails, and structural constraints continuously force models off this nominal path, driving a divergence between benchmark scores and deployment performance. To address this issue, we introduce Decoding-Level Taboo, a zero-prompt diagnostic stress test that intervenes directly in logit space at runtime, forcing models out of their nominal paths. By dynamically masking primary candidate tokens at word boundaries, Taboo forces machine circumlocution. Evaluating Taboo across several open-weight model families reveals that off-path robustness is heavily influenced by both parameter scale and post-training instruction alignment, with robustness generally improving with model size and alignment. Beyond the results presented in this paper, Taboo provides a novel primitive for generating diverse synthetic datasets, stress-testing runtime safety guardrails, and auditing model reliability prior to real-world deployment.

Problem

• LLM robustness
• out-of-distribution generation
• deployment performance
• stress testing
• nominal evaluation

Innovation

• Decoding-Level Taboo
• LLM robustness
• logit-space intervention
• stress testing
• circumlocution