Selecting Offline Reinforcement Learning Algorithms for Stochastic Network Control

2026-03-04

AI Summary

This study addresses the lack of systematic evaluation of offline reinforcement learning (Offline RL) algorithms under realistic stochastic wireless environments characterized by channel fading, noise, and traffic mobility. For the first time, it presents a comprehensive comparison of Bellman-based Conservative Q-Learning (CQL), the sequence modeling approach Decision Transformer (DT), and their hybrid variant Critic-Guided DT within the open-source communication simulation platform mobile-env. Experimental results demonstrate that CQL exhibits the strongest robustness across various stochastic perturbations, making it a reliable default choice, whereas DT-based methods can outperform traditional Bellman approaches when sufficient high-quality trajectories are available. These findings provide empirical evidence and practical guidance for algorithm selection in AI-driven network control systems such as O-RAN and 6G.

Technology Category

• Machine Learning: Online Learning & Bandits
• Multiagent Systems: Multiagent Learning
• Planning, Routing, and Scheduling: Planning/Scheduling and Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Offline Reinforcement Learning (RL) is a promising approach for next-generation wireless networks, where online exploration is unsafe and large amounts of operational data can be reused across the model lifecycle. However, the behavior of offline RL algorithms under genuinely stochastic dynamics -- inherent to wireless systems due to fading, noise, and traffic mobility -- remains insufficiently understood. We address this gap by evaluating Bellman-based (Conservative Q-Learning), sequence-based (Decision Transformers), and hybrid (Critic-Guided Decision Transformers) offline RL methods in an open-access stochastic telecom environment (mobile-env). Our results show that Conservative Q-Learning consistently produces more robust policies across different sources of stochasticity, making it a reliable default choice in lifecycle-driven AI management frameworks. Sequence-based methods remain competitive and can outperform Bellman-based approaches when sufficient high-return trajectories are available. These findings provide practical guidance for offline RL algorithm selection in AI-driven network control pipelines, such as O-RAN and future 6G functions, where robustness and data availability are key operational constraints.

Problem

• Offline Reinforcement Learning
• Stochastic Network Control
• Wireless Networks
• Algorithm Selection
• Robustness

Innovation

• Offline Reinforcement Learning
• Stochastic Network Control
• Conservative Q-Learning
• Decision Transformers
• mobile-env