Joint Beamforming and Resource Allocation for Delay Optimization in RIS-Assisted OFDM Systems: A DRL Approach

2025-06-04

AI Summary

This paper addresses the problem of minimizing average latency in RIS-assisted OFDM downlink systems under stochastic data arrivals. We formulate it as a Markov decision process with a hybrid action space and propose a multi-agent proximal policy optimization (PPO) framework. Our method features a novel dual-branch PPO-Θ/PPO-N architecture that decouples RIS phase-shift control from subcarrier allocation; introduces a delay-sensitive state representation based on buffer queue length and instantaneous arrival rate; designs a distributed subcarrier assignment mechanism to mitigate the curse of dimensionality; and incorporates transfer learning to accelerate convergence. Experimental results demonstrate that the proposed approach reduces average latency by up to 38.2% compared to baseline methods, while simultaneously improving spectral efficiency, user fairness, and robustness and adaptability under dynamic traffic conditions.

Technology Category

• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Multiagent Systems: Mechanism Design

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

This paper investigates a joint phase design and resource allocation problem in downlink reconfigurable intelligent surface (RIS)-assisted orthogonal frequency division multiplexing (OFDM) systems to optimize average delay, where data packets for each user arrive at the base station stochastically. The sequential optimization problem is inherently a Markov decision process (MDP), making it fall within the scope of reinforcement learning. To effectively handle the mixed action space and reduce the state space dimensionality, a hybrid deep reinforcement learning (DRL) approach is proposed. Specifically, proximal policy optimization (PPO)-$Theta$ is employed to optimize RIS phase shift design, while PPO-N is responsible for subcarrier allocation decisions. To further mitigate the curse of dimensionality associated with subcarrier allocation, a multi-agent strategy is introduced to optimize subcarrier allocation indicater more efficiently. Moreover, to achieve more adaptive resource allocation and accurately capture network dynamics, key factors closely related to average delay, including the number of backlogged packets in buffers and the current packet arrivals, are incorporated into the state space. Furthermore, a transfer learning framework is introduced to enhance training efficiency and accelerate convergence. Simulation results demonstrate that the proposed algorithm significantly reduces average delay, enhances resource allocation efficiency, and achieves superior system robustness and fairness compared to baseline methods.

Problem

• Optimize delay in RIS-assisted OFDM systems via joint beamforming and resource allocation
• Address mixed action space and high dimensionality using hybrid DRL approach
• Enhance training efficiency with transfer learning for adaptive resource allocation

Innovation

• Hybrid DRL for RIS phase and subcarrier allocation
• Multi-agent strategy to reduce dimensionality
• Transfer learning to enhance training efficiency