End-to-End Learning-based Video Streaming Enhancement Pipeline: A Generative AI Approach

2025-12-16

AI Summary

Video streaming faces a fundamental trade-off between high visual quality and low latency. Conventional codecs, lacking contextual modeling capabilities, transmit all frame data, resulting in significant bandwidth redundancy. This paper proposes ELVIS, an end-to-end enhancement framework that introduces a novel collaborative paradigm: server-side rate-distortion-optimized encoding coupled with client-side generative reconstruction. Specifically, the server proactively discards redundant frames—e.g., low-motion regions—while the client reconstructs them using diffusion models or transformer-based generative techniques. ELVIS adopts a modular architecture, enabling flexible substitution of encoders, reconstruction models, and quality metrics (e.g., VMAF). Evaluated on standard benchmarks, ELVIS achieves a 11-point VMAF improvement and substantially reduces bandwidth requirements at equivalent perceptual quality. The framework establishes a scalable architectural foundation for integrating generative AI into real-time streaming systems.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

The primary challenge of video streaming is to balance high video quality with smooth playback. Traditional codecs are well tuned for this trade-off, yet their inability to use context means they must encode the entire video data and transmit it to the client. This paper introduces ELVIS (End-to-end Learning-based VIdeo Streaming Enhancement Pipeline), an end-to-end architecture that combines server-side encoding optimizations with client-side generative in-painting to remove and reconstruct redundant video data. Its modular design allows ELVIS to integrate different codecs, inpainting models, and quality metrics, making it adaptable to future innovations. Our results show that current technologies achieve improvements of up to 11 VMAF points over baseline benchmarks, though challenges remain for real-time applications due to computational demands. ELVIS represents a foundational step toward incorporating generative AI into video streaming pipelines, enabling higher quality experiences without increased bandwidth requirements.

Problem

• Balancing video quality with smooth playback
• Reducing redundant data transmission in streaming
• Integrating generative AI for bandwidth efficiency

Innovation

• Server-side encoding optimization reduces redundant video data
• Client-side generative in-painting reconstructs removed video content
• Modular design integrates various codecs and quality metrics