Adaptive TD-Lambda for Cooperative Multi-agent Reinforcement Learning

2026-05-12

AI Summary

This work addresses the challenge in multi-agent reinforcement learning where the vast joint action space and scarce transition data hinder accurate estimation of policy distributions, thereby impeding effective implementation of adaptive TD(λ). To overcome this, the authors propose a novel adaptive TD(λ) method that avoids explicit modeling of policy distributions by employing a parameterized likelihood-free density ratio estimator to dynamically adjust λ. A dual replay buffer architecture is introduced to separately store data from old and new policies, enabling a more precise characterization of the stationary distribution discrepancy over state-action pairs and better balancing the bias-variance trade-off. This approach constitutes the first policy-distribution-free adaptive TD(λ) mechanism in multi-agent settings and is integrated into both QMIX and MAPPO frameworks. Experiments on the SMAC and Google Research Football Academy benchmarks demonstrate consistent performance gains over or parity with fixed-λ baselines.

Abstract

TD($λ$) in value-based MARL algorithms or the Temporal Difference critic learning in Actor-Critic-based (AC-based) algorithms synergistically integrate elements from Monte-Carlo simulation and Q function bootstrapping via dynamic programming, which effectively addresses the inherent bias-variance trade-off in value estimation. Based on that, some recent works link the adaptive $λ$ value to the policy distribution in the single-agent reinforcement learning area. However, because of the large joint action space from multiple number of agents, and the limited transition data in Multi-agent Reinforcement Learning, the policy distribution is infeasible to be calculated statistically. To solve the policy distribution calculation problem in MARL settings, we employ a parametric likelihood-free density ratio estimator with two replay buffers instead of calculating statistically. The two replay buffers of different sizes store the historical trajectories that represent the data distribution of the past and current policies correspondingly. Based on the estimator, we assign Adaptive TD($λ$), \textbf{ATD($λ$)}, values to state-action pairs based on their likelihood under the stationary distribution of the current policy. We apply the proposed method on two competitive baseline methods, QMIX for value-based algorithms, and MAPPO for AC-based algorithms, over SMAC benchmarks and Gfootball academy scenarios, and demonstrate consistently competitive or superior performance compared to other baseline approaches with static $λ$ values.

Problem

• Multi-agent Reinforcement Learning
• policy distribution
• adaptive TD(λ)
• value estimation
• bias-variance trade-off

Innovation

• Adaptive TD(λ)
• Multi-agent Reinforcement Learning
• Density Ratio Estimation
• Likelihood-free Estimation
• Replay Buffer