Improving Fisher Information Estimation and Efficiency for LoRA-based LLM Unlearning

2025-08-28

AI Summary

Large language models (LLMs) risk unintentional leakage of sensitive information, yet full retraining for machine unlearning is prohibitively expensive. Existing methods—e.g., FILA—leverage LoRA for parameter efficiency but still require access to all model parameters and fail to explicitly satisfy the theoretical assumptions underlying Fisher information estimation, leading to biased importance scores. This paper proposes VILA, the first framework to explicitly enforce the foundational assumptions of Fisher information estimation while integrating LoRA for parameter isolation. VILA introduces a novel importance assessment and selective update mechanism that operates without full-parameter access. Evaluated on TOFU, WMDP, and MUSE benchmarks, VILA achieves up to 100× higher parameter efficiency and 40× faster training than FILA, while attaining state-of-the-art unlearning performance.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Learning to Search
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

LLMs have demonstrated remarkable performance across various tasks but face challenges related to unintentionally generating outputs containing sensitive information. A straightforward approach to address this issue is to retrain the model after excluding the problematic data. However, this approach incurs prohibitively high computational costs. To overcome this limitation, machine unlearning has emerged as a promising solution that can effectively remove sensitive information without the need to retrain the model from scratch. Recently, FILA has been proposed as a parameter-efficient unlearning method by integrating LoRA adapters. Specifically, it calculates the Fisher information to identify parameters associated with the forget set and assigns them to LoRA adapters for updates. Despite its innovative approach, FILA still requires access to all model parameters and does not adequately account for fundamental assumptions underlying Fisher information, leading to inaccuracies in importance estimation. To address these limitations, we propose VILA, a novel unlearning framework that explicitly considers the assumptions overlooked in FILA, thereby enhancing the accuracy of parameter identification for the forget set. Moreover, VILA significantly reduces computational costs by enabling parameter identification without accessing the entire model. Our method achieves up to 100x higher parameter efficiency and 40x faster training speed compared to FILA, and sets new state-of-the-art performance on benchmarks including TOFU, WMDP, and MUSE. Our code is available at https://github.com/kyj93790/VILA.

Problem

• Improving Fisher information estimation for LLM unlearning
• Enhancing parameter efficiency without full model access
• Reducing computational costs in sensitive data removal

Innovation

• VILA framework enhances Fisher information accuracy
• Reduces computational costs by partial model access
• Achieves higher parameter efficiency and faster training