Graph Attention Reinforcement Learning for Multicast Routing and Age-Optimal Scheduling

2024-04-28

AI Summary

To address the challenge of Age-of-Information (AoI) optimization in dynamic multicast networks—where tight coupling between routing and scheduling hinders effective control—this paper proposes a hierarchical reinforcement learning (HRL) framework, the first to apply RL to Steiner-tree-based multicast routing. We introduce a normalized graph attention mechanism (NGAT) with contraction-mapping properties, enabling robust graph representation learning and efficient policy optimization. Integrating graph embedding with progressive topology expansion, our method jointly optimizes multicast routing and time-gated multicast scheduling (TGMS). Experiments demonstrate a 11.56× speedup in computational efficiency over conventional algorithms and an AoI approximation ratio of 1.1–1.3. Under energy-constrained conditions, it reduces average weighted AoI and weighted peak AoI by 25.6% and 29.2%, respectively, establishing new performance benchmarks for AoI-aware dynamic multicast control.

Technology Category

• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Age of Information (AoI) has emerged as a prominent metric for evaluating the timeliness of information in time-critical applications, such as video streaming, virtual reality, and metaverse platforms, which often rely on multicast communication. Optimizing AoI in multicast networks is challenging due to the coupling of multicast routing and scheduling decisions, the complexity of the multicast, and the graph representation. This paper focuses on dynamic multicast networks and aims to minimize the expected average AoI by integrating multicast routing and scheduling. To address the inherent complexity of the problem, we first decompose the original problem into two subtasks amenable to hierarchical reinforcement learning (RL) methods. We propose the first RL framework to address the multicast routing problem, also known as the Steiner Tree problem, by incorporating graph embedding and the successive addition of nodes and links. For graph embedding, we propose the Normalized Graph Attention mechanism (NGAT) framework with a proven contraction mapping property, enabling effective graph information capture and superior generalization within the hierarchical RL framework. We validate our framework through experiments on four datasets, including the real-world AS-733 dataset. The results demonstrate that our proposed scheme can be $11.56 imes$ more computationally efficient than traditional multicast routing algorithms while achieving approximation ratios of 1.1-1.3, comparable to state-of-the-art (SOTA) methods. Additionally, our age-optimal TGMS algorithm reduces the average weighted Age of Information (AoI) by 25.6% and the weighted peak age by 29.2% in low-energy scenarios.

Problem

• Optimize Age of Information in dynamic multicast networks
• Integrate multicast routing and scheduling for timeliness
• Address Steiner Tree problem with hierarchical RL

Innovation

• Hierarchical reinforcement learning for multicast routing
• Normalized Graph Attention mechanism (NGAT) framework
• Age-optimal TGMS algorithm reduces AoI significantly