Few-Shot Referring Video Single- and Multi-Object Segmentation via Cross-Modal Affinity with Instance Sequence Matching

2025-04-18

AI Summary

This work addresses few-shot referring video object segmentation (FS-RVOS/RVMOS), tackling challenges in language–vision alignment, referential ambiguity among multiple objects, and cross-frame consistency modeling. We propose a novel method that jointly integrates cross-modal affinity modeling and instance-level temporal matching within a unified Transformer framework. Specifically, we introduce cross-modal attention-driven affinity graph construction and contrastive learning-guided instance sequence matching—both co-optimized end-to-end. A lightweight few-shot adaptation module is further incorporated to enhance generalization with minimal parameters. The approach supports both single- and multi-object scenarios under joint optimization, significantly improving segmentation accuracy and cross-video generalization. Extensive experiments demonstrate state-of-the-art performance across multiple benchmarks, with mAP gains exceeding 5.2% in complex multi-object settings.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Referring video object segmentation (RVOS) aims to segment objects in videos guided by natural language descriptions. We propose FS-RVOS, a Transformer-based model with two key components: a cross-modal affinity module and an instance sequence matching strategy, which extends FS-RVOS to multi-object segmentation (FS-RVMOS). Experiments show FS-RVOS and FS-RVMOS outperform state-of-the-art methods across diverse benchmarks, demonstrating superior robustness and accuracy.

Problem

• Segment objects in videos using language descriptions
• Improve few-shot video object segmentation accuracy
• Extend single-object to multi-object segmentation capability

Innovation

• Transformer-based model for video segmentation
• Cross-modal affinity module enhances accuracy
• Instance sequence matching enables multi-object segmentation