Graph of Agents: Principled Long Context Modeling by Emergent Multi-Agent Collaboration

2025-09-26

AI Summary

To address the limitation of large language models (LLMs) in modeling ultra-long texts due to fixed context windows, this paper proposes a general multi-agent framework that requires no model retraining, architectural modification, or manual prompt engineering. Methodologically, it formalizes long-context modeling as an information-theoretic compression problem and dynamically constructs input-dependent agent collaboration graphs. Integrating retrieval-augmented generation (RAG) with multi-agent coordination, the framework is compatible with arbitrary open-source LLMs (e.g., Llama 3.1-8B, Qwen3-8B). Evaluated on six document question-answering benchmarks, it achieves average F1 gains of +5.7% over standard RAG and +16.35% over fixed-topology multi-agent baselines. Remarkably, with only a 2K-context window, it surpasses the performance of models with 128K-context windows, significantly extending effective context length and cross-task generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Learning
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As a model-agnostic approach to long context modeling, multi-agent systems can process inputs longer than a large language model's context window without retraining or architectural modifications. However, their performance often heavily relies on hand-crafted multi-agent collaboration strategies and prompt engineering, which limit generalizability. In this work, we introduce a principled framework that formalizes the model-agnostic long context modeling problem as a compression problem, yielding an information-theoretic compression objective. Building on this framework, we propose Graph of Agents (GoA), which dynamically constructs an input-dependent collaboration structure that maximizes this objective. For Llama 3.1 8B and Qwen3 8B across six document question answering benchmarks, GoA improves the average $F_1$ score of retrieval-augmented generation by 5.7% and a strong multi-agent baseline using a fixed collaboration structure by 16.35%, respectively. Even with only a 2K context window, GoA surpasses the 128K context window Llama 3.1 8B on LongBench, showing a dramatic increase in effective context length. Our source code is available at https://github.com/tjoo512/graph-of-agents.

Problem

• Multi-agent systems process long inputs without retraining
• Hand-crafted collaboration strategies limit generalizability in current approaches
• Framework formalizes long context modeling as compression problem

Innovation

• Formalizes long context modeling as compression problem
• Dynamically constructs input-dependent agent collaboration structure
• Maximizes information-theoretic objective for multi-agent systems