CMF-IoU: Multi-Stage Cross-Modal Fusion 3D Object Detection with IoU Joint Prediction

2025-08-18

AI Summary

To address the misalignment between 3D geometric space and 2D semantic features in camera–LiDAR multimodal 3D object detection, this paper proposes CMF-IoU, a multi-stage cross-modal fusion framework. Methodologically: (1) depth completion is employed to generate pseudo-point clouds, enabling unified feature representation for images and LiDAR; (2) a dual-branch 3D backbone—comprising S2D (Semantic-to-3D) and ResVC (Residual Voxel-Convolutional) modules—is designed to enhance sparse point cloud modeling and cross-modal consistency; (3) an iterative voxel-and-point-aware pooling mechanism, coupled with joint classification–localization IoU prediction, enables fine-grained proposal refinement. Evaluated on KITTI, nuScenes, and Waymo Open Dataset, CMF-IoU achieves state-of-the-art performance, significantly improving detection accuracy and cross-scenario robustness.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Multi-modal methods based on camera and LiDAR sensors have garnered significant attention in the field of 3D detection. However, many prevalent works focus on single or partial stage fusion, leading to insufficient feature extraction and suboptimal performance. In this paper, we introduce a multi-stage cross-modal fusion 3D detection framework, termed CMF-IOU, to effectively address the challenge of aligning 3D spatial and 2D semantic information. Specifically, we first project the pixel information into 3D space via a depth completion network to get the pseudo points, which unifies the representation of the LiDAR and camera information. Then, a bilateral cross-view enhancement 3D backbone is designed to encode LiDAR points and pseudo points. The first sparse-to-distant (S2D) branch utilizes an encoder-decoder structure to reinforce the representation of sparse LiDAR points. The second residual view consistency (ResVC) branch is proposed to mitigate the influence of inaccurate pseudo points via both the 3D and 2D convolution processes. Subsequently, we introduce an iterative voxel-point aware fine grained pooling module, which captures the spatial information from LiDAR points and textural information from pseudo points in the proposal refinement stage. To achieve more precise refinement during iteration, an intersection over union (IoU) joint prediction branch integrated with a novel proposals generation technique is designed to preserve the bounding boxes with both high IoU and classification scores. Extensive experiments show the superior performance of our method on the KITTI, nuScenes and Waymo datasets.

Problem

• Addresses multi-stage cross-modal fusion for 3D object detection
• Aligns 3D spatial and 2D semantic information from sensors
• Improves bounding box refinement with IoU joint prediction

Innovation

• Multi-stage cross-modal fusion framework
• Bilateral cross-view enhancement backbone
• IoU joint prediction branch integration