Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning

2026-06-22

AI Summary

This work addresses the challenge of efficiently performing selective forgetting or updating historical contributions in LoRA-finetuned models within dynamic decentralized edge networks, where frequent device join/leave events disrupt model consistency. To this end, the authors propose a priority-aware learn-and-forget correction framework that introduces an orthogonal LoRA mechanism to assign each device a contribution coordinate independent of historical dependencies, thereby enabling flexible parameter addition and removal. The framework further integrates a priority-aware correction strategy based on dominant residual terms, along with topology optimization and resource scheduling algorithms, to dynamically determine optimal correction actions. Experimental results demonstrate that the proposed method robustly handles membership changes, adaptively enacts effective corrections across varying residual states, and significantly enhances both the efficiency and stability of model updates.

Abstract

As large language models (LLMs) are increasingly deployed at the network edge to provide pervasive generative AI services, decentralized federated learning (DFL) provides a vital mechanism for privacy-preserving, domain-specific fine-tuning through peer-to-peer exchanges of parameter-efficient updates. However, the dynamic nature of practical decentralized edge networks, where devices may dynamically join or leave the collaborative training process, requires the system to continuously adapt to new data while selectively removing prior contributions. This correction process remains a significant bottleneck, as individual device updates become deeply entangled within the global fine-tuned parameters. To address this challenge, we propose a priority-aware learning-unlearning correction framework based on orthogonal LoRA that can enhance the knowledge evaluation through topology adjustment. Specifically, we first design an orthogonal LoRA mechanism that yields post-training contribution coordinates, enabling history-free projection addition and deletion in response to membership changes. We then analyze the correction bottleneck and develop a priority-aware policy that selects among topology refinement, local correction, proximal damping, and synchronization scheduling according to the dominant residual term. A resource allocation algorithm is further developed to allocate limited communication across layer groups, prioritizing the primary bottlenecks within per-round wireless constraints. Experiments demonstrate that the proposed framework achieves robust post-event correction for both device join and leave events and validate that different residual regimes necessitate distinct correction actions.

Problem

• decentralized federated learning
• dynamic networks
• learning-unlearning
• LoRA fine-tuning
• parameter correction

Innovation

• orthogonal LoRA
• learning-unlearning correction
• priority-aware policy
• decentralized federated learning
• dynamic edge networks