Causal Model-Based Reinforcement Learning for Sample-Efficient IoT Channel Access

2025-11-13

AI Summary

Multi-agent reinforcement learning (MARL) for medium access control (MAC) in IoT suffers from low sample efficiency and opaque, uninterpretable policies. Method: This paper introduces causal learning to IoT MAC control—the first such effort—proposing an interpretable MARL framework grounded in structural causal models (SCMs) and attention mechanisms. It explicitly encodes causal relationships among network variables to enable policy reasoning and causal attribution analysis; integrates data augmentation to generate synthetic trajectories; and employs proximal policy optimization (PPO) for policy training. Contribution/Results: Experiments show the method reduces environment interactions by 58% on average versus black-box MARL baselines, significantly accelerating convergence. Theoretical analysis demonstrates exponential reduction in sample complexity. The framework establishes a new paradigm for deployable, trustworthy distributed control in IoT systems.

Technology Category

• Machine Learning: Causal Learning
• Multiagent Systems: Multiagent Learning
• Reasoning under Uncertainty: Causality

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Despite the advantages of multi-agent reinforcement learning (MARL) for wireless use case such as medium access control (MAC), their real-world deployment in Internet of Things (IoT) is hindered by their sample inefficiency. To alleviate this challenge, one can leverage model-based reinforcement learning (MBRL) solutions, however, conventional MBRL approaches rely on black-box models that are not interpretable and cannot reason. In contrast, in this paper, a novel causal model-based MARL framework is developed by leveraging tools from causal learn- ing. In particular, the proposed model can explicitly represent causal dependencies between network variables using structural causal models (SCMs) and attention-based inference networks. Interpretable causal models are then developed to capture how MAC control messages influence observations, how transmission actions determine outcomes, and how channel observations affect rewards. Data augmentation techniques are then used to generate synthetic rollouts using the learned causal model for policy optimization via proximal policy optimization (PPO). Analytical results demonstrate exponential sample complexity gains of causal MBRL over black-box approaches. Extensive simulations demonstrate that, on average, the proposed approach can reduce environment interactions by 58%, and yield faster convergence compared to model-free baselines. The proposed approach inherently is also shown to provide interpretable scheduling decisions via attention-based causal attribution, revealing which network conditions drive the policy. The resulting combination of sample efficiency and interpretability establishes causal MBRL as a practical approach for resource-constrained wireless systems.

Problem

• Improving sample efficiency in multi-agent reinforcement learning for IoT channel access
• Developing interpretable causal models to replace black-box reinforcement learning approaches
• Reducing environment interactions and achieving faster convergence in wireless systems

Innovation

• Uses structural causal models for interpretable dependencies
• Applies attention-based networks for causal inference
• Employs data augmentation with PPO for policy optimization