Unlearning Works Better Than You Think: Local Reinforcement-Based Selection of Auxiliary Objectives

2025-04-19

AI Summary

In black-box optimization, auxiliary objectives must dynamically adapt to non-stationary landscapes—particularly challenging for non-monotonic Jump functions, where static or passive auxiliary-objective selection fails to exploit stage-specific benefits. Method: We propose the Local Reinforcement-learning-based Auxiliary Objective selection framework (LRSAO) for evolutionary algorithms. LRSAO introduces an active “unlearning” mechanism to discard ineffective auxiliary objectives and employs a local reward design that tightly couples objective selection with real-time landscape characteristics. Contribution/Results: Theoretically, LRSAO achieves black-box complexity Θ(n²/ℓ² + n log n), improving upon the prior best O(n² log n / ℓ). It is the first work to deeply integrate policy-gradient reinforcement learning into evolutionary algorithm frameworks and establishes the first provably efficient general paradigm for dynamic auxiliary-objective management.

Technology Category

• Search and Optimization: Evolutionary Computation
• Machine Learning: Evolutionary Learning
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We introduce Local Reinforcement-Based Selection of Auxiliary Objectives (LRSAO), a novel approach that selects auxiliary objectives using reinforcement learning (RL) to support the optimization process of an evolutionary algorithm (EA) as in EA+RL framework and furthermore incorporates the ability to unlearn previously used objectives. By modifying the reward mechanism to penalize moves that do no increase the fitness value and relying on the local auxiliary objectives, LRSAO dynamically adapts its selection strategy to optimize performance according to the landscape and unlearn previous objectives when necessary. We analyze and evaluate LRSAO on the black-box complexity version of the non-monotonic Jump function, with gap parameter $ell$, where each auxiliary objective is beneficial at specific stages of optimization. The Jump function is hard to optimize for evolutionary-based algorithms and the best-known complexity for reinforcement-based selection on Jump was $O(n^2 log(n) / ell)$. Our approach improves over this result to achieve a complexity of $Theta(n^2 / ell^2 + n log(n))$ resulting in a significant improvement, which demonstrates the efficiency and adaptability of LRSAO, highlighting its potential to outperform traditional methods in complex optimization scenarios.

Problem

• Selects auxiliary objectives using RL for EA optimization
• Improves optimization complexity on non-monotonic Jump functions
• Dynamically adapts and unlearns objectives for better performance

Innovation

• Uses reinforcement learning for objective selection
• Dynamically adapts strategy with local objectives
• Improves optimization complexity significantly