Content Adaptive Encoding For Interactive Game Streaming

2025-11-27

AI Summary

To address the challenge of resolution adaptation under ultra-low latency, zero look-ahead, and no buffering constraints in interactive game streaming (IGS), this paper proposes the first content-adaptive encoding (CAE) framework tailored for IGS. Methodologically, it leverages lightweight block-level statistical features extracted during HEVC encoding to construct a sliding-window data stream, enabling a compact CNN to predict the optimal encoding resolution in real time—achieving dynamic adaptation with zero additional latency. Its key innovation lies in pioneering the application of CAE to IGS while eliminating the latency overhead inherent in conventional approaches that rely on frame-level content analysis or external models. Evaluated on a real-world HEVC streaming system, the framework achieves a 2.3 dB VMAF gain over fixed-resolution baselines; each inference takes only 1 ms, introducing no computational or transmission latency.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Video-on-demand streaming has benefitted from extit{content-adaptive encoding} (CAE), i.e., adaptation of resolution and/or quantization parameters for each scene based on convex hull optimization. However, CAE is very challenging to develop and deploy for interactive game streaming (IGS). Commercial IGS services impose ultra-low latency encoding with no lookahead or buffering, and have extremely tight compute constraints for any CAE algorithm execution. We propose the first CAE approach for resolution adaptation in IGS based on compact encoding metadata from past frames. Specifically, we train a convolutional neural network (CNN) to infer the best resolution from the options available for the upcoming scene based on a running window of aggregated coding block statistics from the current scene. By deploying the trained CNN within a practical IGS setup based on HEVC encoding, our proposal: (i) improves over the default fixed-resolution ladder of HEVC by 2.3 Bjøntegaard Delta-VMAF points; (ii) infers using 1ms of a single CPU core per scene, thereby having no latency overhead.

Problem

• Develops content-adaptive encoding for interactive game streaming
• Uses past frame metadata to infer optimal resolution per scene
• Achieves quality gains with minimal latency and compute overhead

Innovation

• CNN predicts optimal resolution from past frame metadata
• Uses aggregated coding block statistics for real-time adaptation
• Deploys within HEVC encoding with minimal latency overhead