Beyond Attack Success Rate: Temporal Logit Observability for LLM Safety Failures

2026-05-28

AI Summary

Traditional attack success rate (ASR) provides only a binary outcome, failing to capture the dynamic process of safety failures in large language models (LLMs). This work proposes a training-free diagnostic method that, for the first time, enables observability of when and how safety failures occur by analyzing the evolution of the compliance-rejection boundary during decoding, using only logits. Leveraging Temporal Logit Observability (TLO), marginal analysis, and logit sequence modeling, the approach maps model–attack pairs onto an interpretable two-dimensional plane and supports early intervention. Experiments across four aligned LLMs and three jailbreaking paradigms demonstrate that attacks with identical ASR can be effectively distinguished, and a TLO-based early-stopping strategy reduces successful jailbreaks by over 50% without false positives on benign queries.

Abstract

Attack Success Rate (ASR) evaluates each jailbreak with a single yes/no label at the end of generation, telling us whether a failure happened but not how it unfolded. Two attacks that produce equally harmful outputs may have followed completely different paths, and ASR cannot tell them apart. We make those hidden paths observable from logits alone. Temporal Logit Observability (TLO) is a training-free diagnostic that watches a compliance-refusal margin during decoding and places each model-attack condition on a calibrated 2D plane. By design, this plane is most informative exactly where ASR is least informative: among attacks that succeed for genuinely different reasons. Across four aligned LLMs and three jailbreak paradigms, attacks with nearly identical ASR land at clearly different points on the plane: the same model can fail through different temporal patterns. The geometry matches refusal-direction probes from hidden states on most conditions, with one model showing the limit of our fixed-lexicon approach. A simple early-stop rule derived from TLO cuts successful jailbreaks by more than half, without false alarms on plain benign queries. Safety evaluation should report when and how a failure unfolds, not only whether it occurred. TLO makes the first two observable from logits alone.

Problem

• Attack Success Rate
• LLM Safety
• Temporal Logit Observability
• Jailbreak
• Safety Evaluation

Innovation

• Temporal Logit Observability
• LLM safety
• jailbreak detection
• training-free diagnostic
• compliance-refusal margin