Mind the Gap: Learning Modality-Agnostic Representations with a Cross-Modality UNet

2026-05-16

AI Summary

This work addresses the challenge in cross-modal recognition where modality discrepancies often lead to loss of discriminative information or failed transformations, thereby limiting matching performance and generalization. To mitigate this, the authors propose cmUNet, an encoder–decoder architecture that jointly learns modality-invariant yet identity-preserving representations through cross-modal transformation and within-modality reconstruction. Adversarial and perceptual losses are incorporated to enhance indistinguishability in the original data space. Building upon this, a MarrNet matching framework is developed. Notably, the study introduces occlusion robustness as a novel metric to evaluate the ability to bridge modality gaps. Extensive experiments on Raman–infrared spectral matching, cross-modal person re-identification, and heterogeneous face recognition demonstrate that the proposed method significantly outperforms existing approaches, validating its effectiveness and strong generalization capability.

Abstract

Cross-modality recognition has many important applications in science, law enforcement and entertainment. Popular methods to bridge the modality gap include reducing the distributional differences of representations of different modalities, learning indistinguishable representations or explicit modality transfer. The first two approaches suffer from the loss of discriminant information while removing the modality-specific variations. The third one heavily relies on the successful modality transfer, could face catastrophic performance drop when explicit modality transfers are not possible or difficult. To tackle this problem, we proposed a compact encoder-decoder neural module (cmUNet) to learn modality-agnostic representations while retaining identity-related information. This is achieved through cross-modality transformation and in-modality reconstruction, enhanced by an adversarial/perceptual loss which encourages indistinguishability of representations in the original sample space. For cross-modality matching, we propose MarrNet where cmUNet is connected to a standard feature extraction network which takes as inputs the modality-agnostic representations and outputs similarity scores for matching. We validated our method on five challenging tasks, namely Raman-infrared spectrum matching, cross-modality person re-identification and heterogeneous (photo-sketch, visible-near infrared and visible-thermal) face recognition, where MarrNet showed superior performance compared to state-of-the-art methods. Furthermore, it is observed that a cross-modality matching method could be biased to extract discriminant information from partial or even wrong regions, due to incompetence of dealing with modality gaps, which subsequently leads to poor generalization. We show that robustness to occlusions can be an indicator of whether a method can well bridge the modality gap.

Problem

• cross-modality recognition
• modality gap
• modality-agnostic representations
• discriminant information loss
• generalization

Innovation

• modality-agnostic representation
• cross-modality UNet
• adversarial/perceptual loss
• cmUNet
• MarrNet