Multi-Resolution Alignment for Voxel Sparsity in Camera-Based 3D Semantic Scene Completion

2026-02-03

IEEE Transactions on Image Processing

AI Summary

This work addresses the optimization challenges in camera-based 3D semantic scene completion arising from sparse voxel representations. To mitigate this issue, the authors propose a multi-resolution view Transformer that jointly models scene-level and instance-level semantic consistency across scales by integrating a cubic semantic anisotropy module and a key distribution alignment module. The approach leverages multi-resolution 3D feature projection, scene feature fusion, cubic neighborhood semantic discrepancy analysis, and a cyclic distribution alignment loss to effectively alleviate sparsity-induced optimization difficulties. Experimental results demonstrate that the proposed method significantly improves both completion accuracy and training efficiency. The implementation has been made publicly available.

Technology Category

• Computer Vision: 3D Computer Vision
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Camera-based 3D semantic scene completion (SSC) offers a cost-effective solution for assessing the geometric occupancy and semantic labels of each voxel in the surrounding 3D scene with image inputs, providing a voxel-level scene perception foundation for the perception-prediction-planning autonomous driving systems. Although significant progress has been made in existing methods, their optimization rely solely on the supervision from voxel labels and face the challenge of voxel sparsity as a large portion of voxels in autonomous driving scenarios are empty, which limits both optimization efficiency and model performance. To address this issue, we propose a Multi-Resolution Alignment (MRA) approach to mitigate voxel sparsity in camera-based 3D semantic scene completion, which exploits the scene and instance level alignment across multi-resolution 3D features as auxiliary supervision. Specifically, we first propose the Multi-resolution View Transformer module, which projects 2D image features into multi-resolution 3D features and aligns them at the scene level through fusing discriminative seed features. Furthermore, we design the Cubic Semantic Anisotropy module to identify the instance-level semantic significance of each voxel, accounting for the semantic differences of a specific voxel against its neighboring voxels within a cubic area. Finally, we devise a Critical Distribution Alignment module, which selects critical voxels as instance-level anchors with the guidance of cubic semantic anisotropy, and applies a circulated loss for auxiliary supervision on the critical feature distribution consistency across different resolutions. Extensive experiments on the SemanticKITTI and SSCBench-KITTI-360 datasets demonstrate that our MRA approach significantly outperforms existing state-of-the-art methods, showcasing its effectiveness in mitigating the impact of sparse voxel labels. The code is available at https://github.com/PKU-ICST-MIPL/MRA_TIP.

Problem

• voxel sparsity
• 3D semantic scene completion
• camera-based perception
• autonomous driving
• scene completion

Innovation

• Multi-Resolution Alignment
• Voxel Sparsity
• Semantic Scene Completion
• Cubic Semantic Anisotropy
• Critical Distribution Alignment