NegoCollab: A Common Representation Negotiation Approach for Heterogeneous Collaborative Perception

2025-10-31

AI Summary

To address feature-domain shift arising from heterogeneous agents employing disparate fixed perception models in collaborative perception, this paper proposes a negotiation-based shared representation learning framework. Unlike existing approaches that anchor shared representations on a single agent—introducing alignment bias—we introduce a learnable negotiator that dynamically synthesizes a unified, multimodal-aware representation. Furthermore, we design a bidirectional sender–receiver transformation architecture that jointly optimizes distributional, structural, and semantic (pragmatic) alignment losses to enable end-to-end knowledge distillation. Experiments demonstrate that our method significantly reduces domain gaps across multiple heterogeneous collaborative perception benchmarks, consistently improving collaborative detection performance. Notably, it achieves high training efficiency, model-agnostic compatibility, and seamless integration with diverse backbone architectures.

Technology Category

• Multiagent Systems: Agreement, Argumentation & Negotiation
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Representation Learning for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Collaborative perception improves task performance by expanding the perception range through information sharing among agents. . Immutable heterogeneity poses a significant challenge in collaborative perception, as participating agents may employ different and fixed perception models. This leads to domain gaps in the intermediate features shared among agents, consequently degrading collaborative performance. Aligning the features of all agents to a common representation can eliminate domain gaps with low training cost. However, in existing methods, the common representation is designated as the representation of a specific agent, making it difficult for agents with significant domain discrepancies from this specific agent to achieve proper alignment. This paper proposes NegoCollab, a heterogeneous collaboration method based on the negotiated common representation. It introduces a negotiator during training to derive the common representation from the local representations of each modality's agent, effectively reducing the inherent domain gap with the various local representations. In NegoCollab, the mutual transformation of features between the local representation space and the common representation space is achieved by a pair of sender and receiver. To better align local representations to the common representation containing multimodal information, we introduce structural alignment loss and pragmatic alignment loss in addition to the distribution alignment loss to supervise the training. This enables the knowledge in the common representation to be fully distilled into the sender.

Problem

• Addresses domain gaps in heterogeneous collaborative perception systems
• Proposes negotiated common representation to align diverse agent features
• Enables effective knowledge transfer between local and shared representations

Innovation

• Negotiated common representation for heterogeneous agents
• Sender-receiver pairs transform features between spaces
• Multi-loss alignment for structural and pragmatic training