Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning

2025-06-04

AI Summary

Harmful fine-tuning (HFT) systematically degrades the safety alignment of open-source large language models, yet existing defenses overlook the heterogeneous vulnerability inherent in alignment data. This work is the first to identify and model a systematic, cross-task forgettable vulnerable subset within alignment data. Building on this insight, we propose a vulnerability-aware group-robust optimization framework that jointly integrates vulnerability estimation, adversarial sampling, and group-dependent adversarial perturbation to achieve balanced robust learning across vulnerable groups. Evaluated on four HFT benchmark tasks, our method significantly reduces harmfulness (average reduction of 38.2%) while preserving downstream task performance—outperforming all existing state-of-the-art approaches comprehensively.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Harmful fine-tuning (HFT), performed directly on open-source LLMs or through Fine-tuning-as-a-Service, breaks safety alignment and poses significant threats. Existing methods aim to mitigate HFT risks by learning robust representation on alignment data or making harmful data unlearnable, but they treat each data sample equally, leaving data vulnerability patterns understudied. In this work, we reveal that certain subsets of alignment data are consistently more prone to forgetting during HFT across different fine-tuning tasks. Inspired by these findings, we propose Vulnerability-Aware Alignment (VAA), which estimates data vulnerability, partitions data into"vulnerable"and"invulnerable"groups, and encourages balanced learning using a group distributionally robust optimization (Group DRO) framework. Specifically, VAA learns an adversarial sampler that samples examples from the currently underperforming group and then applies group-dependent adversarial perturbations to the data during training, aiming to encourage a balanced learning process across groups. Experiments across four fine-tuning tasks demonstrate that VAA significantly reduces harmful scores while preserving downstream task performance, outperforming state-of-the-art baselines.

Problem

• Mitigates uneven forgetting in harmful fine-tuning of LLMs
• Identifies vulnerable alignment data prone to safety breaches
• Balances learning using adversarial sampling and perturbations

Innovation

• Estimates data vulnerability patterns
• Partitions data into vulnerable groups
• Uses Group DRO for balanced learning