Executable Knowledge Graphs for Replicating AI Research

2025-10-20

AI Summary

Large language model (LLM)-based agents struggle to faithfully reproduce AI research due to insufficient background knowledge, the inability of standard RAG to capture implicit technical details in papers, and the lack of multi-granular, executable knowledge representations—leading to non-executable code. Method: This paper proposes a modular, pluggable executable knowledge graph (xKG), which integrates technical insights, code snippets, and domain-specific knowledge from scientific literature. xKG enables fine-grained retrieval and cross-task reuse, and is constructed end-to-end via joint information extraction, structured knowledge modeling, and RAG enhancement. Contribution/Results: Integrated with three mainstream agent frameworks and two LLM families on the PaperBench benchmark, xKG achieves up to a 10.9% absolute improvement in execution success rate, demonstrating its effectiveness and generality in enhancing reproducibility, deepening technical understanding, and ensuring code executability.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Knowledge Representation and Reasoning: Knowledge Acquisition
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Replicating AI research is a crucial yet challenging task for large language model (LLM) agents. Existing approaches often struggle to generate executable code, primarily due to insufficient background knowledge and the limitations of retrieval-augmented generation (RAG) methods, which fail to capture latent technical details hidden in referenced papers. Furthermore, previous approaches tend to overlook valuable implementation-level code signals and lack structured knowledge representations that support multi-granular retrieval and reuse. To overcome these challenges, we propose Executable Knowledge Graphs (xKG), a modular and pluggable knowledge base that automatically integrates technical insights, code snippets, and domain-specific knowledge extracted from scientific literature. When integrated into three agent frameworks with two different LLMs, xKG shows substantial performance gains (10.9% with o3-mini) on PaperBench, demonstrating its effectiveness as a general and extensible solution for automated AI research replication. Code will released at https://github.com/zjunlp/xKG.

Problem

• Addressing challenges in replicating AI research using LLM agents
• Overcoming limitations in generating executable code from papers
• Integrating technical insights and code snippets from scientific literature

Innovation

• Executable Knowledge Graphs integrate technical insights and code
• xKG captures latent technical details from scientific literature
• Modular knowledge base supports multi-granular retrieval and reuse