Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Data

2025-05-15

AI Summary

This work exposes severe security risks introduced by pseudo-malicious data fine-tuning in large language models (LLMs) for cybersecurity, including degraded prompt injection resistance, heightened personal data leakage vulnerability, and increased propensity to generate malicious code. To address this, we propose a safety-alignment method based on rewriting instruction–response pairs, explicitly embedding security constraints and ethical principles while preserving core technical capabilities. Leveraging the OWASP LLM Top 10 framework, we systematically evaluate seven mainstream open-source models (e.g., Llama 3.1 8B, Qwen 2.5 7B). Results show that fine-tuning reduces prompt injection defense scores by over 80% on average (e.g., Llama 3.1 8B drops from 0.95 to 0.15). Our method fully restores—often exceeding—original safety scores, significantly enhancing model robustness and regulatory compliance without sacrificing utility.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Security

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The integration of large language models (LLMs) into cyber security applications presents significant opportunities, such as enhancing threat analysis and malware detection, but can also introduce critical risks and safety concerns, including personal data leakage and automated generation of new malware. We present a systematic evaluation of safety risks in fine-tuned LLMs for cyber security applications. Using the OWASP Top 10 for LLM Applications framework, we assessed seven open-source LLMs: Phi 3 Mini 3.8B, Mistral 7B, Qwen 2.5 7B, Llama 3 8B, Llama 3.1 8B, Gemma 2 9B, and Llama 2 70B. Our evaluation shows that fine-tuning reduces safety resilience across all tested LLMs (e.g., the safety score of Llama 3.1 8B against prompt injection drops from 0.95 to 0.15). We propose and evaluate a safety alignment approach that carefully rewords instruction-response pairs to include explicit safety precautions and ethical considerations. This approach demonstrates that it is possible to maintain or even improve model safety while preserving technical utility, offering a practical path forward for developing safer fine-tuning methodologies. This work offers a systematic evaluation for safety risks in LLMs, enabling safer adoption of generative AI in sensitive domains, and contributing towards the development of secure, trustworthy, and ethically aligned LLMs.

Problem

• Evaluating safety risks in fine-tuned LLMs for cyber security
• Assessing reduced safety resilience post fine-tuning in LLMs
• Proposing safety alignment to balance utility and ethical concerns

Innovation

• Evaluated safety risks in fine-tuned LLMs systematically
• Proposed safety alignment via reworded instruction-response pairs
• Maintained model safety while preserving technical utility