Reinforcement Learning for Dynamic Resource Allocation in Optical Networks: Hype or Hope?

2025-02-18

AI Summary

This work addresses critical challenges in reinforcement learning (RL) research for dynamic resource allocation in optical networks—namely, the lack of standardized benchmarks, poor reproducibility, and questionable performance advantages. We introduce an open-source, topology-agnostic simulation framework with standardized evaluation protocols. To enable theoretically grounded assessment, we propose a fragmentation-aware analytical lower bound on blocking probability. Systematic comparisons are conducted between state-of-the-art RL methods and lightweight heuristic algorithms (e.g., path counting/sorting). Experimental results across diverse topologies reveal that the best heuristics achieve blocking probabilities one order of magnitude lower than leading RL approaches; moreover, RL yields at most 19–36% gain in admissible traffic load, with diminishing returns fundamentally constrained by network fragmentation characteristics. Our findings indicate limited marginal utility of current RL solutions in practical optical networks, underscoring the necessity of reproducible benchmarking and theory-informed evaluation frameworks.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

The application of reinforcement learning (RL) to dynamic resource allocation in optical networks has been the focus of intense research activity in recent years, with almost 100 peer-reviewed papers. We present a review of progress in the field, and identify significant gaps in benchmarking practices and reproducibility. To determine the strongest benchmark algorithms, we systematically evaluate several heuristics across diverse network topologies. We find that path count and sort criteria for path selection significantly affect the benchmark performance. We meticulously recreate the problems from five landmark papers and apply the improved benchmarks. Our comparisons demonstrate that simple heuristics consistently match or outperform the published RL solutions, often with an order of magnitude lower blocking probability. Furthermore, we present empirical lower bounds on network blocking using a novel defragmentation-based method, revealing that potential improvements over the benchmark heuristics are limited to 19--36% increased traffic load for the same blocking performance in our examples. We make our simulation framework and results publicly available to promote reproducible research and standardized evaluation https://doi.org/10.5281/zenodo.12594495.

Problem

• Evaluate reinforcement learning in optical networks
• Identify gaps in benchmarking and reproducibility
• Compare heuristics with RL solutions

Innovation

• Reinforcement Learning application
• Systematic heuristic evaluation
• Novel defragmentation-based method