Adaptive Design of mmWave Initial Access Codebooks using Reinforcement Learning

2025-10-29

AI Summary

In 5G millimeter-wave initial access, static Synchronization Signal Block (SSB) beam codebooks struggle to adapt to dynamic user distributions. To address this, we propose a reinforcement learning (RL)-driven adaptive codebook design method integrated with domain-specific expert knowledge. The approach requires no prior user location information and dynamically optimizes SSB beam direction and width via real-time environmental sensing, while respecting hardware-imposed physical constraints. Its key innovation lies in the first integration of expert knowledge into an RL framework, yielding an interpretable and deployable hybrid beam management mechanism. Experimental results demonstrate that, compared to conventional expert-designed configurations, the proposed method improves average user access success rate by 10.8%, significantly enhancing coverage reliability and environmental adaptability in complex heterogeneous scenarios.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Mixture of Experts (MoE)
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Initial access (IA) is the process by which user equipment (UE) establishes its first connection with a base station. In 5G systems, particularly at millimeter-wave frequencies, IA integrates beam management to support highly directional transmissions. The base station employs a codebook of beams for the transmission of Synchronization Signal Blocks (SSBs), which are periodically swept to detect and connect users. The design of this SSB codebook is critical for ensuring reliable, wide-area coverage. In current networks, SSB codebooks are meticulously engineered by domain experts. While these expert-defined codebooks provide a robust baseline, they lack flexibility in dynamic or heterogeneous environments where user distributions vary, limiting their overall effectiveness. This paper proposes a hybrid Reinforcement Learning (RL) framework for adaptive SSB codebook design. Building on top of expert knowledge, the RL agent leverages a pool of expert-designed SSB beams and learns to adaptively select or combine them based on real-time feedback. This enables the agent to dynamically tailor codebooks to the actual environment, without requiring explicit user location information, while always respecting practical beam constraints. Simulation results demonstrate that, on average, the proposed approach improves user connectivity by 10.8$%$ compared to static expert configurations. These findings highlight the potential of combining expert knowledge with data-driven optimization to achieve more intelligent, flexible, and resilient beam management in next-generation wireless networks.

Problem

• Adaptive SSB codebook design using reinforcement learning
• Overcoming static codebook limitations in dynamic environments
• Improving user connectivity without location information

Innovation

• Reinforcement Learning adapts SSB codebook design dynamically
• Agent combines expert-designed beams using real-time feedback
• Method improves connectivity without location information requirements