ContextNav: Towards Agentic Multimodal In-Context Learning

2025-10-06

AI Summary

Current multimodal in-context learning (ICL) methods struggle to balance scalability and robustness: manually curated demonstrations achieve high quality but lack scalability, while similarity-based retrieval scales well yet often introduces noisy or structurally inconsistent examples. To address this, we propose ContextNav—the first agent-based framework that unifies scalable automatic retrieval with human-like curation quality. Its core contributions are threefold: (1) the first integration of agent mechanisms into multimodal context management; (2) a closed-loop, operation-syntax-graph-driven workflow that jointly optimizes retrieval, cross-modal alignment, and context construction in an adaptive manner; and (3) the integration of resource-aware embeddings, retrievable vector stores, and graph-based orchestration. ContextNav achieves state-of-the-art performance across multiple benchmarks, significantly enhancing both the stability of multimodal ICL under noisy conditions and its generalization capability.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent advances demonstrate that multimodal large language models (MLLMs) exhibit strong multimodal in-context learning (ICL) capabilities, enabling them to adapt to novel vision-language tasks from a few contextual examples. However, existing ICL approaches face challenges in reconciling scalability with robustness across diverse tasks and noisy contextual examples: manually selecting examples produces clean contexts but is labor-intensive and task-specific, while similarity-based retrieval improves scalability but could introduce irrelevant or structurally inconsistent samples that degrade ICL performance. To address these limitations, we propose ContextNav, the first agentic framework that integrates the scalability of automated retrieval with the quality and adaptiveness of human-like curation, enabling noise-robust and dynamically optimized contextualization for multimodal ICL. ContextNav unifies context management and noise-robust contextualization within a closed-loop workflow driven by graph-based orchestration. Specifically, it builds a resource-aware multimodal embedding pipeline, maintains a retrievable vector database, and applies agentic retrieval and structural alignment to construct noise-resilient contexts. An Operational Grammar Graph (OGG) further supports adaptive workflow planning and optimization, enabling the agent to refine its operational strategies based on downstream ICL feedback. Experimental results demonstrate that ContextNav achieves state-of-the-art performance across various datasets, underscoring the promise of agentic workflows for advancing scalable and robust contextualization in multimodal ICL.

Problem

• Addresses scalability and robustness challenges in multimodal in-context learning
• Reconciles automated retrieval quality with human-like curation for noisy contexts
• Enables dynamic optimization of contextualization through agentic workflow integration

Innovation

• Agentic framework combining automated retrieval with human-like curation
• Graph-based orchestration for unified context management and noise-robust contextualization
• Operational Grammar Graph enabling adaptive workflow optimization via feedback