MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

2024-07-29

arXiv.org

AI Summary

To address three key challenges in complex information retrieval and cross-page knowledge integration—(1) query semantic complexity rendering single-step retrieval ineffective, (2) critical information scattered across high-noise web pages, and (3) long documents exceeding large language model (LLM) context windows—this paper proposes WebPlanner-WebSearcher, a multi-agent framework inspired by human cognition. It enables dynamic subproblem decomposition, hierarchical web page retrieval, and parallel information fusion. Crucially, it introduces a human-like collaborative planning mechanism that supports concurrent processing of over 300 web pages, overcoming both single-retrieval and context-length limitations. Experiments demonstrate that the framework completes information integration equivalent to three hours of manual effort within three minutes, significantly improving answer depth and breadth on both open- and closed-set QA benchmarks. The InternLM2.5-7B implementation outperforms ChatGPT-Web and Perplexity.ai.

Technology Category

• Multiagent Systems: Multiagent Planning
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Information seeking and integration is a complex cognitive task that consumes enormous time and effort. Inspired by the remarkable progress of Large Language Models, recent works attempt to solve this task by combining LLMs and search engines. However, these methods still obtain unsatisfying performance due to three challenges: (1) complex requests often cannot be accurately and completely retrieved by the search engine once (2) corresponding information to be integrated is spread over multiple web pages along with massive noise, and (3) a large number of web pages with long contents may quickly exceed the maximum context length of LLMs. Inspired by the cognitive process when humans solve these problems, we introduce MindSearch to mimic the human minds in web information seeking and integration, which can be instantiated by a simple yet effective LLM-based multi-agent framework. The WebPlanner models the human mind of multi-step information seeking as a dynamic graph construction process: it decomposes the user query into atomic sub-questions as nodes in the graph and progressively extends the graph based on the search result from WebSearcher. Tasked with each sub-question, WebSearcher performs hierarchical information retrieval with search engines and collects valuable information for WebPlanner. The multi-agent design of MindSearch enables the whole framework to seek and integrate information parallelly from larger-scale (e.g., more than 300) web pages in 3 minutes, which is worth 3 hours of human effort. MindSearch demonstrates significant improvement in the response quality in terms of depth and breadth, on both close-set and open-set QA problems. Besides, responses from MindSearch based on InternLM2.5-7B are preferable by humans to ChatGPT-Web and Perplexity.ai applications, which implies that MindSearch can already deliver a competitive solution to the proprietary AI search engine.

Problem

• Addresses complex information retrieval challenges in web searches
• Overcomes limitations of single-query search engine performance
• Solves context length constraints when processing multiple web pages

Innovation

• LLM-based multi-agent framework mimics human cognition
• Dynamic graph construction for multi-step information seeking
• Hierarchical information retrieval from large-scale web pages