FROC: A Unified Framework with Risk-Optimized Control for Machine Unlearning in LLMs

2025-12-15

AI Summary

Existing machine unlearning methods for large language models (LLMs) lack rigorous quantification, trade-off analysis, and controllable mitigation of the dual risks of insufficient forgetting and utility degradation. Method: We propose the first conformal inference–based, risk-controllable unlearning framework. It introduces a configurable continuous risk model and the first data-driven forgetting risk estimator (CUR), formalizing unlearning as a probabilistic risk-budget–constrained process. By integrating semantic shift modeling, risk-sensitive optimization, and multi-objective hyperparameter validation, we design a joint forgetting-utility scoring function. Results: Evaluated across multiple state-of-the-art unlearning methods, our framework consistently characterizes interpretable risk–utility Pareto frontiers. It ensures sufficient forgetting while significantly improving model generalization—providing both theoretical foundations and practical tools for trustworthy implementation of the “right to be forgotten.”

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Machine unlearning (MU) seeks to eliminate the influence of specific training examples from deployed models. As large language models (LLMs) become widely used, managing risks arising from insufficient forgetting or utility loss is increasingly crucial. Current MU techniques lack effective mechanisms for evaluating and controlling these risks, hindering the selection of strategies that appropriately balance safety and utility, and raising trust concerns surrounding the "right to be forgotten." To address these issues, we propose FROC, a unified framework with Risk-Optimized Control for machine unlearning in LLMs. FROC is built around a conformal-style risk-control formulation that expresses a user-specified risk budget on unlearning behavior. This probability-based constraint enables FROC to compare MU strategies, identify feasible operating regions, and guide hyperparameter selection according to desired trade-offs between forgetting sufficiency and utility preservation. To operationalize this constraint, FROC introduces a smoothly varying continuous risk model that aggregates forgetting deficiency and utility degradation into a single configuration-level score. Building on conformal risk analysis, FROC computes (1) the Conformal Unlearning Risk (CUR), a data-driven estimated value on the probability that forgotten samples continue to influence model predictions, and (2) risk-controlled configuration sets, which identify unlearning hyperparameters that are valid under the specified risk budget. Experiments across multiple LLM MU methods demonstrate that FROC produces stable, interpretable risk landscapes and reveals consistent relationships between unlearning configurations, semantic shift, and utility impact. FROC reframes MU as a controllable, risk-aware process and offers a practical foundation for managing unlearning behavior in large-scale LLM deployments.

Problem

• Develops a framework for evaluating and controlling risks in machine unlearning for LLMs
• Addresses the lack of mechanisms to balance forgetting sufficiency and utility preservation
• Provides a risk-aware process to manage unlearning behavior in large-scale LLM deployments

Innovation

• Framework with conformal risk-control for unlearning
• Continuous risk model aggregating forgetting and utility metrics
• Data-driven CUR and risk-controlled configuration sets