Decoding-Level Taboo: A Diagnostic Stress Test for LLM Robustness
This work addresses the performance degradation of large language models (LLMs) in real-world deployment, where system prompts or safety constraints often divert generation from standard pathways despite strong benchmark results. To diagnose such robustness gaps, the authors propose a prompt-free stress-testing methodology that dynamically intervenes in the logit space during decoding by masking dominant candidate tokens at word boundaries, thereby compelling the model to generate via alternative, non-canonical paths. This approach introduces a novel runtime dynamic token-masking mechanism, establishing a zero-prompt stress-testing paradigm. Experiments across multiple open-source LLMs reveal significant effects of model scale and instruction alignment on robustness under such perturbations. Furthermore, the method efficiently produces diverse synthetic data, enabling pre-deployment reliability auditing and evaluation of safety mechanisms.