RAS: Measuring LLM Safety Through Refusal Alignment

2026-06-24

AI Summary

Current safety evaluations of large language models predominantly rely on output-level judgments, which suffer from high computational costs, strong subjectivity, and poor generalizability. This work proposes SafeVec, a white-box evaluation method that introduces refusal alignment as an internal representational signal for the first time. By extracting inter-layer refusal directions from a reference model and measuring the alignment between a target model’s hidden states and this direction within a stable layer window, SafeVec produces a 0–100 Refusal Alignment Score (RAS) without requiring any output generation. This approach substantially improves evaluation efficiency and accurately distinguishes aligned from misaligned models across Llama, Gemma, and Qwen model families. Moreover, the RAS exhibits a strong correlation with attack success rates, outperforming conventional black-box methods based on external evaluators.

Abstract

Safety evaluation of large language models (LLMs) is commonly performed by querying models with unsafe or jailbreak prompts and judging whether their outputs violate a safety policy. Although useful, output-level evaluation is expensive, sensitive to judge choice, and easily tied to fixed question banks. We propose **SafeVec**, a white-box evaluation procedure that measures safety from internal representations rather than generated answers. **SafeVec** first extracts layer-wise refusal directions from a safety-aligned reference model, then selects stable layer windows where safe and unsafe behaviors are separable, and finally scores a target model by measuring whether its hidden states align with these refusal directions under unsafe and jailbreak prompts. The resulting metric, **RAS** (**R**efusal **A**lignment **S**core), maps representation-level refusal alignment to a calibrated 0-100 safety score. Across `Llama`, `Gemma`, and `Qwen` model families, RAS separates aligned models from uncensored and abliterated variants, tracks output-level attack success rate, and is substantially faster than judge-based evaluation. These results suggest that refusal alignment provides a compact and efficient signal for white-box LLM safety evaluation.

Problem

• LLM safety evaluation
• output-level evaluation
• jailbreak prompts
• safety policy
• refusal alignment

Innovation

• SafeVec
• Refusal Alignment Score
• white-box evaluation
• LLM safety
• internal representations