Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models
In black-box LLM-as-a-Service (LLMaaS) settings, stealthy backdoor alignment attacks—where models violate safety alignment upon inputs containing hidden triggers—are notoriously difficult to detect. Method: We propose BEAT, the first sample-agnostic, black-box detectable defense leveraging distortions in refusal signals. Its core insight is the “probe concatenation effect”: a stable, significant drop in refusal rate upon backdoor activation. Instead of analyzing output semantics, BEAT monitors the stability of safety signals via multi-sample output distribution estimation, probe concatenation perturbations, and KL-divergence–based distortion quantification—requiring neither gradients nor internal model access. Results: Evaluated on closed- and open-source models including GPT-3.5-turbo, BEAT achieves AUC > 0.96 in detecting diverse backdoor attacks and generalizes effectively against mainstream jailbreak techniques.