NeRFCom: Feature Transform Coding Meets Neural Radiance Field for Free-View 3D Scene Semantic Transmission

2025-02-27

IEEE Communications Letters

AI Summary

Traditional NeRF-based semantic transmission for free-viewpoint 3D scenes suffers from bandwidth inefficiency and poor error resilience due to the decoupled design of NeRF compression and channel coding. To address this, we propose an end-to-end jointly optimized framework integrating neural radiance field (NeRF) semantic modeling, deep learning-driven nonlinear transform coding, and a learnable probabilistic entropy model—enabling semantic-aware, variable-bitrate joint source–channel coding that dynamically allocates bits according to semantic importance. This work is the first to co-model semantic-aware reconstruction quality and channel characteristics, ensuring high-fidelity free-viewpoint rendering even under adverse channel conditions. Experimental results demonstrate an average PSNR gain of 2.8 dB over conventional approaches, significantly improving both transmission efficiency and robustness.

Technology Category

• Computer Vision: 3D Computer Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

We introduce NeRFCom, a novel communication system designed for end-to-end 3D scene transmission. Compared to traditional systems relying on handcrafted NeRF semantic feature decomposition for compression and well-adaptive channel coding for transmission error correction, our NeRFCom employs a nonlinear transform and learned probabilistic models, enabling flexible variable-rate joint source-channel coding and efficient bandwidth allocation aligned with the NeRF semantic feature's different contribution to the 3D scene synthesis fidelity. Experimental results demonstrate that NeRFCom achieves free-view 3D scene efficient transmission while maintaining robustness under adverse channel conditions.

Problem

• End-to-end 3D scene transmission
• Flexible variable-rate joint source-channel coding
• Efficient bandwidth allocation for NeRF semantic features

Innovation

• Neural Radiance Field
• Nonlinear Transform Coding
• Joint Source-Channel Coding