EfficientPENet: Real-Time Depth Completion from Sparse LiDAR via Lightweight Multi-Modal Fusion

2026-04-20

AI Summary

This work addresses the challenge of efficient depth completion from sparse LiDAR point clouds and RGB images by proposing a lightweight dual-branch network. Built upon ConvNeXt as the backbone, the method integrates sparsity-invariant convolutions with a Convolutional Spatial Propagation Network (CSPN), employing late fusion and multi-scale depth supervision. It is the first to introduce layer normalization, large-kernel depthwise convolutions, and stochastic depth regularization into depth completion. Additionally, a position-aware test-time augmentation strategy is devised. On the KITTI benchmark, the model achieves an RMSE of 631.94 mm with only 36.24M parameters and a latency of 20.51 ms (48.76 FPS), reducing the parameter count by 3.7× and accelerating inference by 23× compared to BP-Net.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Non-convex Optimization
• Machine Learning: Learning on the Edge & Model Compression

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Depth completion from sparse LiDAR measurements and corresponding RGB images is a prerequisite for accurate 3D perception in robotic systems. Existing methods achieve high accuracy on standard benchmarks but rely on heavy backbone architectures that preclude real-time deployment on embedded hardware. We present EfficientPENet, a two-branch depth completion network that replaces the conventional ResNet encoder with a modernized ConvNeXt backbone, introduces sparsity-invariant convolutions for the depth stream, and refines predictions through a Convolutional Spatial Propagation Network (CSPN). The RGB branch leverages ImageNet-pretrained ConvNeXt blocks with Layer Normalization, 7x7 depthwise convolutions, and stochastic depth regularization. Features from both branches are merged via late fusion and decoded through a multi-scale deep supervision strategy. We further introduce a position-aware test-time augmentation scheme that corrects coordinate tensors during horizontal flipping, yielding consistent error reduction at inference. On the KITTI depth completion benchmark, EfficientPENet achieves an RMSE of 631.94 mm with 36.24M parameters and a latency of 20.51 ms, operating at 48.76 FPS. This represents a 3.7 times reduction in parameters and a 23 times speedup relative to BP-Net, while maintaining competitive accuracy. These results establish EfficientPENet as a practical solution for real-time depth completion on resource-constrained edge platforms such as the NVIDIA Jetson.

Problem

• depth completion
• sparse LiDAR
• real-time
• embedded hardware
• multi-modal fusion

Innovation

• EfficientPENet
• ConvNeXt
• sparsity-invariant convolution
• Convolutional Spatial Propagation Network
• test-time augmentation