Towards Agentic AI for Multimodal-Guided Video Object Segmentation

2025-08-14

AI Summary

Reference-based video object segmentation (RVOS) is a multimodal task where existing methods either rely on costly supervised training or suffer from inflexible, fixed pipelines lacking dynamic adaptability. Method: This paper introduces the first training-free, agent-based RVOS framework: leveraging a large language model (LLM) to dynamically orchestrate a workflow that synergistically integrates vision-language foundation models, multimodal understanding modules, and low-level visual tools for cross-frame, fine-grained object localization and segmentation. Contribution/Results: The framework abandons predefined pipelines, instead adapting its strategy autonomously based on input referring expressions. It achieves performance on par with or surpassing fully supervised, task-specific models on both the RVOS and Ref-AVS benchmarks, demonstrating significant improvements in flexibility and accuracy—particularly in complex, diverse scenarios.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Referring-based Video Object Segmentation is a multimodal problem that requires producing fine-grained segmentation results guided by external cues. Traditional approaches to this task typically involve training specialized models, which come with high computational complexity and manual annotation effort. Recent advances in vision-language foundation models open a promising direction toward training-free approaches. Several studies have explored leveraging these general-purpose models for fine-grained segmentation, achieving performance comparable to that of fully supervised, task-specific models. However, existing methods rely on fixed pipelines that lack the flexibility needed to adapt to the dynamic nature of the task. To address this limitation, we propose Multi-Modal Agent, a novel agentic system designed to solve this task in a more flexible and adaptive manner. Specifically, our method leverages the reasoning capabilities of large language models (LLMs) to generate dynamic workflows tailored to each input. This adaptive procedure iteratively interacts with a set of specialized tools designed for low-level tasks across different modalities to identify the target object described by the multimodal cues. Our agentic approach demonstrates clear improvements over prior methods on two multimodal-conditioned VOS tasks: RVOS and Ref-AVS.

Problem

• Multimodal-guided video object segmentation lacks flexibility
• Traditional methods require high computational and annotation costs
• Fixed pipelines fail to adapt to dynamic task demands

Innovation

• Agentic system for flexible multimodal segmentation
• LLM-driven dynamic workflow generation
• Specialized tools for low-level multimodal tasks