The JPEG Pleno Learning-based Point Cloud Coding Standard: Serving Man and Machine

2024-09-12

arXiv.org

AI Summary

This work addresses the challenge of jointly optimizing point cloud compression efficiency and downstream machine understanding in the compressed domain. We propose the JPEG Pleno Learned Point Cloud Compression (PCC) standard—an ISO/IEC international standard featuring a novel dual-objective unified framework: native 3D sparse convolutional networks for geometry coding, and joint learning of 2D projection-based color coding with JPEG AI. The framework enables end-to-end joint optimization and compressed-domain feature extraction. Compared to MPEG PCC, it achieves significantly lower geometric bitrates and superior reconstruction quality. Furthermore, we establish the first benchmark explicitly designed for dual-purpose evaluation—assessing both human visual perception and AI tasks (e.g., detection, segmentation)—and validate its practicality and generalizability across VR and autonomous driving applications. This standard represents the first internationally standardized deep learning–based point cloud compression framework that simultaneously optimizes for human vision and machine intelligence.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Learning on the Edge & Model Compression
• Data Mining & Knowledge Management: Data Compression

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Efficient point cloud coding has become increasingly critical for multiple applications such as virtual reality, autonomous driving, and digital twin systems, where rich and interactive 3D data representations may functionally make the difference. Deep learning has emerged as a powerful tool in this domain, offering advanced techniques for compressing point clouds more efficiently than conventional coding methods while also allowing effective computer vision tasks performed in the compressed domain thus, for the first time, making available a common compressed visual representation effective for both man and machine. Taking advantage of this potential, JPEG has recently finalized the JPEG Pleno Learning-based Point Cloud Coding (PCC) standard offering efficient lossy coding of static point clouds, targeting both human visualization and machine processing by leveraging deep learning models for geometry and color coding. The geometry is processed directly in its original 3D form using sparse convolutional neural networks, while the color data is projected onto 2D images and encoded using the also learning-based JPEG AI standard. The goal of this paper is to provide a complete technical description of the JPEG PCC standard, along with a thorough benchmarking of its performance against the state-of-the-art, while highlighting its main strengths and weaknesses. In terms of compression performance, JPEG PCC outperforms the conventional MPEG PCC standards, especially in geometry coding, achieving significant rate reductions. Color compression performance is less competitive but this is overcome by the power of a full learning-based coding framework for both geometry and color and the associated effective compressed domain processing.

Problem

• Develops efficient point cloud coding for VR, autonomous driving, and digital twins.
• Introduces deep learning-based JPEG Pleno standard for human and machine use.
• Benchmarks JPEG PCC against MPEG standards, highlighting compression improvements.

Innovation

• Deep learning for efficient point cloud compression
• Sparse CNNs for 3D geometry processing
• JPEG AI for 2D color data encoding