PF3Det: A Prompted Foundation Feature Assisted Visual LiDAR 3D Detector

2025-04-04

AI Summary

Addressing the challenges of cross-modal feature alignment and scarce high-quality annotations in LiDAR-camera multimodal 3D object detection for autonomous driving, this paper proposes a low-data-dependency detection framework synergistically driven by foundation models and soft prompts. Methodologically, it integrates pretrained vision (CLIP) and point cloud (Point-BERT) encoders with learnable soft prompts to enable semantic alignment and feature enhancement across modalities. Additionally, it introduces cross-modal feature distillation and a lightweight fusion head to improve robustness and computational efficiency. Evaluated on the nuScenes benchmark using limited annotated data, the framework achieves state-of-the-art performance: +1.19% in NuScenes Detection Score (NDS) and +2.42% in mean Average Precision (mAP), demonstrating superior data efficiency and generalization without requiring extensive supervision.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

3D object detection is crucial for autonomous driving, leveraging both LiDAR point clouds for precise depth information and camera images for rich semantic information. Therefore, the multi-modal methods that combine both modalities offer more robust detection results. However, efficiently fusing LiDAR points and images remains challenging due to the domain gaps. In addition, the performance of many models is limited by the amount of high quality labeled data, which is expensive to create. The recent advances in foundation models, which use large-scale pre-training on different modalities, enable better multi-modal fusion. Combining the prompt engineering techniques for efficient training, we propose the Prompted Foundational 3D Detector (PF3Det), which integrates foundation model encoders and soft prompts to enhance LiDAR-camera feature fusion. PF3Det achieves the state-of-the-art results under limited training data, improving NDS by 1.19% and mAP by 2.42% on the nuScenes dataset, demonstrating its efficiency in 3D detection.

Problem

• Efficiently fusing LiDAR and camera data for 3D detection
• Overcoming domain gaps between LiDAR and image modalities
• Reducing reliance on large labeled datasets for training

Innovation

• Integrates foundation model encoders
• Uses soft prompts for feature fusion
• Enhances LiDAR-camera fusion efficiently