HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models

2026-07-13

AI Summary

Fine-tuning large language models often compromises their safety alignment, increasing the risk of generating harmful outputs. Existing defense mechanisms typically suffer from high computational costs, strong intrusiveness, or poor generalization. To address these limitations, this work proposes HyperSafe—a non-intrusive, inference-time safety recovery framework that dynamically generates a dedicated Safety Side Network (SSN) for any fine-tuned model via a hypernetwork. Leveraging layer activation fingerprints, HyperSafe identifies and blocks harmful requests in a single forward pass without modifying the original model or requiring retraining. Experiments on Qwen2-7B and LLaMA-3-8B demonstrate that HyperSafe reduces harmful response rates from 19–31% to below 1%, while incurring an average degradation of no more than 1% in downstream task accuracy.

Abstract

Safety alignment in large language models can be fragile under fine-tuning, as even benign task adaptation may increase harmful compliance. Existing defenses mainly follow two directions: they either intervene during or after fine-tuning through retraining or weight modification, which can be costly and may hurt task performance, or they use model-agnostic safety classifiers, which may miss failures specific to a given fine-tuned checkpoint. These limitations motivate a post hoc, model-specific, and non-invasive approach to safety restoration. To meet these requirements, we propose HyperSafe, a framework that restores safety behavior by generating a model-specific Safe Side Network (SSN) for each fine-tuned checkpoint. HyperSafe uses layer-wise activation fingerprints to capture how fine-tuning changes the model's inner representations. With a small set of given calibration prompts, the hypernetwork maps these fingerprints to the parameters of the \ssn{} in a single forward pass. The generated \ssn{} runs alongside the frozen fine-tuned model and performs prompt-level safety classification: harmful prompts are routed to refusal, while safe prompts are answered by the original fine-tuned model. Thus, HyperSafe requires no gradient updates, no safety data at deployment time, and no modification to the deployed model weights. We evaluate HyperSafe on two model families, Qwen2-7B and LLaMA-3-8B, across multiple safety benchmarks. HyperSafe reduces harmful response rates from 19-31% to below 1% on every held-out checkpoint, while keeping downstream task accuracy within 1% of the fine-tuned baseline on average. Code is available at https://github.com/nokronim/project-safety-remedy.

Problem

• safety alignment
• fine-tuning
• harmful compliance
• language models
• safety recovery

Innovation

• HyperSafe
• Safe Side Network
• inference-time safety
• activation fingerprint
• non-invasive safety recovery