Randomized Antipodal Search Done Right for Data Pareto Improvement of LLM Unlearning

2026-04-17

AI Summary

This work addresses the challenge of unlearning undesirable knowledge from large language models (LLMs) in practical scenarios where complete datasets for both forgetting and retention are often unavailable, thereby limiting the effectiveness of existing unlearning methods. To tackle this issue, the study introduces, for the first time, a formalization of data Pareto improvement within the LLM unlearning framework and proposes the Random Antipodal Search with Linearized Influence Kernels (RASLIK) algorithm. By integrating permutation-projection hashing with a random antipodal sampling strategy, RASLIK enables efficient data selection with sublinear complexity and low variance. Theoretical analysis and extensive experiments demonstrate that RASLIK consistently outperforms baseline approaches across diverse models, datasets, and unlearning settings—achieving a superior trade-off between effective forgetting and knowledge retention—and even surpasses oracle-based sampling in certain cases.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) sometimes memorize undesirable knowledge, which must be removed after deployment. Prior work on machine unlearning has focused largely on optimization methods that adjust parameters to enforce forgetting while preserving retention. However, these approaches assume that the forget and retain sets are readily available, which rarely holds in practice. Unlearning is typically triggered by an undesired generation at inference time, making the retrieval of relevant data the central challenge. We introduce the notion of data Pareto improvement for LLM unlearning, which formalizes how retrieval can expand the achievable trade-off frontier between forgetting and retention. To realize this principle, we propose Randomized Antipodal Search on Linearized Influence Kernel (RASLIK), a retrieval algorithm that combines permutation-projection hashing with randomized antipodal search. RASLIK reduces selection variance, achieves sublinear complexity, and yields a double gain in both quality and efficiency. Across multiple models, datasets, and unlearning algorithms, RASLIK consistently outperforms deterministic baselines and even oracle sampling, establishing randomized search as a principled and scalable solution for data-centric unlearning.

Problem

• machine unlearning
• data retrieval
• large language models
• Pareto improvement
• undesirable knowledge

Innovation

• data Pareto improvement
• randomized antipodal search
• LLM unlearning
• influence-based retrieval
• sublinear complexity