High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning

2025-10-23

AI Summary

In multi-agent reinforcement learning (MARL), modeling high-order interactions often incurs combinatorial explosion, while black-box value decomposition hinders interpretable credit assignment. To address these challenges, we propose QCoFr—a continued-fraction-based framework for interpretable value decomposition. Its core innovation lies in the first-ever explicit modeling of arbitrary-order cooperative relationships with linear computational complexity, coupled with a variational information bottleneck to extract robust latent credit signals. Unlike existing methods, QCoFr avoids combinatorial explosion while significantly enhancing decision transparency and robustness to observation noise. Empirical evaluation across multiple benchmark tasks demonstrates superior performance and strong alignment between learned credit assignments and theoretical expectations—validating its effectiveness, interpretability, and stability.

Technology Category

• Multiagent Systems: Adversarial Agents
• Reasoning under Uncertainty: Sequential Decision Making
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The ability to model interactions among agents is crucial for effective coordination and understanding their cooperation mechanisms in multi-agent reinforcement learning (MARL). However, previous efforts to model high-order interactions have been primarily hindered by the combinatorial explosion or the opaque nature of their black-box network structures. In this paper, we propose a novel value decomposition framework, called Continued Fraction Q-Learning (QCoFr), which can flexibly capture arbitrary-order agent interactions with only linear complexity $mathcal{O}left({n} ight)$ in the number of agents, thus avoiding the combinatorial explosion when modeling rich cooperation. Furthermore, we introduce the variational information bottleneck to extract latent information for estimating credits. This latent information helps agents filter out noisy interactions, thereby significantly enhancing both cooperation and interpretability. Extensive experiments demonstrate that QCoFr not only consistently achieves better performance but also provides interpretability that aligns with our theoretical analysis.

Problem

• Model high-order agent interactions efficiently
• Avoid combinatorial explosion in multi-agent learning
• Enhance cooperation interpretability via information bottleneck

Innovation

• Continued Fraction Q-Learning for flexible high-order interactions
• Linear complexity modeling to avoid combinatorial explosion
• Variational information bottleneck for interpretable credit estimation