SAMOT: State-Aware Step Modulation and Optimal Transport Matching for Audio-Visual Instance Segmentation

2026-08-08

AI Summary

This work addresses the challenges of long-term modeling and instance association in audio-visual instance segmentation, which arise from dynamic modality states and cross-modal distribution discrepancies. To tackle these issues, the authors propose an adaptive dynamic step-size modulation mechanism and an explicit matching strategy based on optimal transport. The former dynamically adjusts the Mamba step size by integrating temporal evolution, cross-modal divergence, and historical context, enabling rapid response to sounding objects while ensuring stable modeling. The latter jointly optimizes cross-modal distribution alignment and structural correspondence through entropy-regularized optimal transport and maximum mean discrepancy (MMD) regularization. Evaluated on the AVIS benchmark, the proposed method achieves significant improvements, with gains of 3.76, 2.75, and 2.58 in FSLA, HOTA, and mAP, respectively, demonstrating its effectiveness and state-of-the-art performance.

Abstract

Audio-Visual Instance Segmentation (AVIS) aims to simultaneously classify, segment, and track sounding objects within video sequences. Unlike Audio-Visual Semantic Segmentation (AVS), AVIS involves instance-level modeling across longer video sequences, introducing two key challenges: (1) complex modality-state changes disrupt long-range modeling, and (2) substantial structural and distributional discrepancies between modalities hinder precise instance-level association. Existing methods rely on fixed-step Transformers and recursive Mamba models, lacking adaptability to modality-state changes. In addition, methods performing implicit matching ignore the inherent distributional inconsistencies. To address these issues, we propose a framework with Adaptive Dynamic Step Modulation (ADSM) and Optimal Transport-based Matching Modulation (OT-MM). ADSM adaptively modulates Mamba step sizes using temporal variation, cross-modal discrepancy, and historical context, balancing rapid response to modality-state changes with stable long-range modeling. OT-MM explicitly formulates instance-level cross-modal matching as an entropy-regularized optimal transport problem solved via log-domain Sinkhorn iterations, and further enforces distribution-level coherence with an MMD regularizer. Extensive experiments demonstrate state-of-the-art performance on the AVIS benchmark (+3.76 FSLA, +2.75 HOTA, +2.58 mAP), verified through comprehensive qualitative visualizations. The code and model are available at https://github.com/happylife-pk/SAMOT.

Problem

• Audio-Visual Instance Segmentation
• modality-state changes
• cross-modal discrepancy
• long-range modeling
• instance-level association

Innovation

• Adaptive Dynamic Step Modulation
• Optimal Transport
• Audio-Visual Instance Segmentation
• Cross-modal Matching
• Mamba