Offline to Online Learning for Real-Time Bandwidth Estimation

2023-09-23

AI Summary

Personalized bandwidth estimation (BWE) for real-time video communication over heterogeneous networks suffers from poor adaptability and customization. Method: We propose a knowledge-informed, data-driven neural BWE framework. First, we neurosymbolically reconstruct classical heuristic BWE algorithms and employ behavior cloning on offline operational logs to learn user-level policies—enabling personalized initialization without online exploration. Then, we integrate lightweight incremental online fine-tuning to close the loop for end-to-end bandwidth control optimization. Contributions/Results: (1) We pioneer the “heuristic neurosymbolization + behavior cloning transfer” paradigm, eliminating manual hyperparameter tuning. (2) In real intercontinental video conferences, our method achieves baseline heuristic-level QoE and improves it by 7.8% after fine-tuning. (3) Compared to state-of-the-art online reinforcement learning approaches, it reduces the number of required call sessions for convergence by 80%.

Abstract

Real-time video applications require accurate bandwidth estimation (BWE) to maintain user experience across varying network conditions. However, increasing network heterogeneity challenges general-purpose BWE algorithms, necessitating solutions that adapt to end-user environments. While widely adopted, heuristic-based methods are difficult to individualize without extensive domain expertise. Conversely, online reinforcement learning (RL) offers ease of customization but neglects prior domain expertise and suffers from sample inefficiency. Thus, we present Merlin, an imitation learning-based solution that replaces the manual parameter tuning of heuristic-based methods with data-driven updates to streamline end-user personalization. Our key insight is that transforming heuristic-based BWE algorithms into neural networks facilitates data-driven personalization. Merlin utilizes Behavioral Cloning to efficiently learn from offline telemetry logs, capturing heuristic policies without live network interactions. The cloned policy can then be seamlessly tailored to end user network conditions through online finetuning. In real intercontinental videoconferencing calls, Merlin matches our heuristic's policy with no statistically significant differences in user quality of experience (QoE). Finetuning Merlin's control policy to end-user environments enables QoE improvements of up to 7.8% compared to the heuristic policy. Lastly, our IL-based design performs competitively with current state-of-the-art online RL techniques but converges with 80% fewer videoconferencing samples, facilitating practical end-user personalization.

Problem

• Adapts bandwidth estimation to user environments
• Replaces heuristic tuning with data-driven updates
• Improves user experience with fewer samples

Innovation

• Imitation learning replaces heuristic tuning
• Behavioral Cloning learns from offline data
• Online finetuning improves user experience