Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis

2026-07-07

AI Summary

This study addresses the trade-off between safety alignment and utility in vulnerability analysis within large language models (LLMs), a challenge exacerbated by difficulties in cross-model attribution. To enable controlled evaluation, the authors construct paired models from the same family—differing only in the presence (Aligned) or absence (Abliterated) of safety refusal mechanisms—and systematically assess their performance across vulnerability detection, CWE attribution, line-level localization, and executable patch generation. Experiments on Gemma and Qwen model series using the Vul4J dataset demonstrate that Abliterated models substantially enhance practical utility: Gemma’s patch usability (67.8% vs. 29.9%) and compilation success rate (32.8% vs. 9.0%) markedly improve, while Qwen achieves doubled F1 and Top-1 accuracy in line-level localization. These findings underscore the necessity of jointly evaluating response rate, correctness, and actionability in LLM-based security tasks.

Abstract

Large language model (LLM)-assisted software security operates at a difficult boundary: the vulnerability-analysis terminology needed for legitimate code review, triage, and repair can closely resemble terminology associated with misuse. Existing safety and cybersecurity evaluations are difficult to interpret in this setting because they often compare unrelated model families, thereby conflating safety behavior with differences in architecture, scale, training data, and deployment. To isolate this factor, we study safety state: whether refusal behavior remains intact (Aligned) or has been refusal-ablated (Abliterated) within same-lineage models. We ask how this safety state affects defensive utility across software-security workflows. We compare aligned instruction-tuned models with publicly released refusal-ablated descendants from two model families, Gemma and Qwen. We evaluate Aligned and Abliterated states on vulnerability detection, CWE attribution, vulnerable-line localization, root-cause localization, and executable patch validation. We further treat prompt wording as a controlled framing dimension: prompts begin with neutral code-review language, add authorization context, and vary the density of cybersecurity terminology. In a Gemma-based Java/Vul4J repair-validation study, Abliterated achieves higher early-stage validation rates, with 67.8%, 65.0%, and 32.8% of patches judged usable, successfully applied, and successfully compiled, respectively, compared with 29.9%, 24.9%, and 9.0% for Aligned. In the Qwen pair, Abliterated improves localization performance, increasing line-level F1 from 2.08% to 3.91% and Top-1 accuracy from 4.10% to 6.95%. These findings suggest that evaluations of LLM-based security assistants should jointly measure whether models respond, whether their usable responses are correct, and whether their outputs remain actionable across the engineering workflow.

Problem

• large language models
• software security
• vulnerability analysis
• refusal behavior
• safety evaluation

Innovation

• same-lineage study
• refusal ablation
• LLM safety alignment
• vulnerability analysis
• prompt framing