Stagnant Neuron: Towards Understanding the Plasticity Loss in Multi-Agent Reinforcement Learning Value Factorization Methods

2026-06-23

AI Summary

This work addresses the challenge of diminished plasticity in value decomposition methods for multi-agent reinforcement learning when transferring to new tasks, often caused by neuronal stagnation that impedes continual learning. To overcome this limitation, the authors propose KNIFE, a novel approach that intervenes directly at the neuron level by identifying stagnant units through gradient magnitude analysis and replacing them with a composite structure designed to preserve prior knowledge, restore neuronal activity, and compensate for output discrepancies. This unified mechanism effectively balances the retention of learned collaborative knowledge with the recovery of learning capacity. Empirical evaluations demonstrate that KNIFE significantly outperforms existing plasticity-injection techniques across multiple benchmarks—including SMACv2, predator-prey environments, and matrix games—yielding substantial improvements in transfer adaptability.

Abstract

Multi-Agent Reinforcement Learning (MARL) value factorization methods can suffer from a loss of plasticity, gradually failing to adapt when transferring to new task instances. We trace this issue to stagnant neurons, units whose gradient updates become negligibly small relative to their weights, thereby hindering learning. While existing plasticity injection methods exist, they prove ineffective for such neurons. To address this, we propose Knowledge-retentive Neuron-level PlastIcity Focusing InjEction (KNIFE), a novel method that directly targets stagnant neurons. KNIFE replaces each stagnant neuron with a composite unit comprising three specialized components: a frozen knowledge neuron to preserve acquired knowledge, a re-initialized active neuron to restore learning capacity, and a compensation neuron to ensure the combined output matches the original, thus maintaining previous learned cooperation knowledge. Extensive experiments on SMACv2, predator-prey, and matrix games demonstrate that KNIFE significantly outperforms state-of-the-art plasticity injection methods.

Problem

• plasticity loss
• stagnant neuron
• multi-agent reinforcement learning
• value factorization
• adaptation

Innovation

• stagnant neuron
• plasticity loss
• value factorization
• multi-agent reinforcement learning
• KNIFE