SkillGraph: Self-Evolving Multi-Agent Collaboration with Multimodal Graph Topology

2026-04-19

AI Summary

Existing vision-based multi-agent systems typically rely on fixed communication topologies and static reasoning capabilities, limiting their ability to dynamically adapt collaborative strategies to visual content and query context. This work proposes SkillGraph, a framework that jointly enables the self-evolution of both communication topology and agent skills. It replaces handcrafted routing mechanisms with a multimodal graph transformer that constructs query-aware dynamic collaboration graphs, and introduces a skill distillation module that extracts generalizable reasoning strategies from failure cases to continuously update a multimodal skill repository. Extensive experiments demonstrate consistent performance gains across four benchmarks, five multi-agent architectures, and four foundation models.

Technology Category

• Multiagent Systems: Multiagent Learning
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Scaling vision-language models into Visual Multiagent Systems (VMAS) is hindered by two coupled issues. First, communication topologies are fixed before inference, leaving them blind to visual content and query context; second, agent reasoning abilities remain static during deployment. These issues reinforce each other: a rigid topology fails to leverage richer agent expertise, while static agents lack incentives to specialize for a given query. We address this with SkillGraph, a joint framework that evolves both agent expertise and communication topology. Within this framework, a Multimodal Graph Transformer (MMGT) encodes visual tokens, instruction semantics and active skill embeddings to predict a query-conditioned collaboration graph, replacing hand-crafted routing with dynamic, content-aware information flow. Complementing this, a Skill Designer distills and refines reasoning heuristics from failure cases, constructing a self-evolving multimodal Skill Bank. Crucially, updated skill embeddings are fed back into the MMGT, enabling the topology to adapt alongside capability growth. Experiments show that SkillGraph achieves consistent improvements across four benchmarks, five common MAS structures and four base models. Code is available at https://github.com/niez233/skillgraph.

Problem

• Visual Multiagent Systems
• communication topology
• agent reasoning
• dynamic adaptation
• multimodal

Innovation

• Multimodal Graph Transformer
• Self-Evolving Skill Bank
• Dynamic Collaboration Topology
• Visual Multiagent Systems
• Query-Conditioned Routing