R-ACP: Real-Time Adaptive Collaborative Perception Leveraging Robust Task-Oriented Communications

2024-10-05

arXiv.org

AI Summary

To address the dual challenges of dynamic extrinsic parameter errors and communication timeliness in multi-robot collaborative perception, this paper proposes a real-time adaptive collaborative perception framework. Methodologically, it introduces the Age of Perceived Targets (AoPT)—a novel timeliness metric replacing the conventional Age of Information (AoI); proposes channel-aware, cross-camera re-identification–driven online self-calibration; and designs an information-bottleneck–based semantic video compression network with priority-aware feature filtering. The key contributions lie in the first unified modeling of perception-target timeliness, robust re-identification, and semantic compression. Under adverse channel conditions, the framework achieves a 25.49% improvement in Multi-Object Detection Accuracy (MODA) and reduces communication overhead by 51.36%, significantly outperforming five state-of-the-art baseline methods.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Multiagent Systems: Agent Communication

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Collaborative perception enhances sensing in multirobot and vehicular networks by fusing information from multiple agents, improving perception accuracy and sensing range. However, mobility and non-rigid sensor mounts introduce extrinsic calibration errors, necessitating online calibration, further complicated by limited overlap in sensing regions. Moreover, maintaining fresh information is crucial for timely and accurate sensing. To address calibration errors and ensure timely and accurate perception, we propose a robust task-oriented communication strategy to optimize online self-calibration and efficient feature sharing for Real-time Adaptive Collaborative Perception (R-ACP). Specifically, we first formulate an Age of Perceived Targets (AoPT) minimization problem to capture data timeliness of multi-view streaming. Then, in the calibration phase, we introduce a channel-aware self-calibration technique based on reidentification (Re-ID), which adaptively compresses key features according to channel capacities, effectively addressing calibration issues via spatial and temporal cross-camera correlations. In the streaming phase, we tackle the trade-off between bandwidth and inference accuracy by leveraging an Information Bottleneck (IB) based encoding method to adjust video compression rates based on task relevance, thereby reducing communication overhead and latency. Finally, we design a priority-aware network to filter corrupted features to mitigate performance degradation from packet corruption. Extensive studies demonstrate that our framework outperforms five baselines, improving multiple object detection accuracy (MODA) by 25.49% and reducing communication costs by 51.36% under severely poor channel conditions. Code will be made publicly available: github.com/fangzr/R-ACP.

Problem

• Addresses extrinsic calibration errors in multi-agent perception systems
• Ensures timely data fusion for accurate real-time perception
• Optimizes bandwidth usage while maintaining high inference accuracy

Innovation

• Robust task-oriented communication optimizes online self-calibration
• Channel-aware self-calibration via adaptive feature compression
• Information Bottleneck encoding adjusts video compression rates