HeteroPROPMT: A Real-time and Privacy-Preserving Heterogeneous Collaborative Perception Framework

2026-07-28

AI Summary

This work addresses the challenge of inconsistent feature spaces among heterogeneous agents—stemming from differences in sensors, models, and training domains—which severely limits collaborative perception performance. Existing approaches suffer from poor scalability and privacy concerns. To overcome these limitations, the authors propose a lightweight, privacy-preserving alignment framework that modulates bird’s-eye-view (BEV) features via modular visual prompts while keeping each participant’s encoder and fusion detection modules frozen. By integrating an autoencoder-driven modality identification and routing mechanism, the method aligns heterogeneous features to the ego-centric coordinate space in real time without requiring metadata. The approach introduces only a minimal number of trainable parameters, yet achieves significant gains in average precision on OPV2V-H and V2XSet benchmarks, with modality identification accuracy exceeding 99.99% and trainable parameters reduced by several orders of magnitude.

Abstract

Collaborative Perception (CP) improves autonomous systems' awareness of their surroundings by sharing sensor data, intermediate features, and detection results. In real-world deployments, however, collaborating vehicles often use heterogeneous sensors, perception models, datasets, and training domains, creating feature-space shifts that degrade downstream fusion and detection. Existing approaches typically retrain fusion and detection components or introduce modality-specific feature interpreters. These methods scale poorly to newly joining agents and often require access to proprietary metadata, raising privacy concerns. We propose HeteroPROMPT, a real-time and privacy-preserving framework for heterogeneous collaborative perception. HeteroPROMPT rapidly aligns each heterogeneous agent's features with an ego-centric unified feature space through modular prompts and lightweight learning-based tuning, while keeping agent encoders and the collaborative fusion and detection stacks frozen. Its visual prompt-based training and inference modulate Bird's Eye View (BEV) features across channels and spatial locations with low computational overhead. For metadata-free deployment, an autoencoder learns a compact unified representation and extracts modality cues from shared features, enabling real-time modality classification and routing to the appropriate HeteroPROMPT modules without exposing proprietary agent information. Experiments on the OPV2V-H and V2XSet datasets show that HeteroPROMPT improves Average Precision over state-of-the-art heterogeneous CP methods while using orders of magnitude fewer trainable parameters. This offers a scalable and practical CP solution. The proposed modality classifier also predicts the joining agent's modality from compact features with greater than 99.99 percent accuracy during deployment. Code will be available at https://github.com/arminmaleki007/HeteroPROMPT.

Problem

• Collaborative Perception
• Heterogeneous Sensors
• Feature-space Shift
• Privacy Preservation
• Scalability

Innovation

• Heterogeneous Collaborative Perception
• Visual Prompting
• Privacy-Preserving
• Modality Classification
• Feature Alignment