63‐4: A Novel Real‐Time Full‐Color 3D Holographic (Diffractive) Video Capture, Processing, and Transmission Pipeline Using Off‐the‐Shelf Hardware

2022-06-01

SID Symposium Digest of Technical Papers

AI Summary

This work presents the first real-time, full-color, dynamic 3D holographic video communication system deployed entirely on off-the-shelf commercial hardware, addressing the core bottlenecks of computational intensity and bandwidth constraints in real-time holographic telepresence. Methodologically, we propose an end-to-end pipeline comprising RGBZ acquisition, diffraction computation, lightweight compression, and network transmission: leveraging multi-modal iPhone sensors (RGB + depth) for 3D scene capture, VividQ’s holographic SDK for real-time phase optimization and diffraction pattern generation, and a custom efficient encoding scheme with adaptive transmission protocol. The system achieves 720p@30fps full-color 3D holographic video calls with end-to-end latency under 120 ms, enabling deployment on consumer-grade devices and cross-platform display. Crucially, it requires no custom optical or computational hardware—establishing the first reproducible, scalable pathway toward practical holographic communication.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Hardware-aware ML
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This paper details the world's first live 3D holographic (diffractive) video call using off-the-shelf hardware. We introduce a novel pipeline that facilitates the capture, processing, and transmission of RGBZ data, using an iPhone for image and depth capture with VividQ's SDK for hologram generation and hardware for display.

Problem

• Real-time 3D holographic video call
• Capture, process, transmit RGBZ data
• Use off-the-shelf hardware for implementation

Innovation

• Real-time holographic video
• Off-the-shelf hardware
• RGBZ data processing