What Papers Don't Tell You: Recovering Tacit Knowledge for Automated Paper Reproduction

2026-03-02

AI Summary

This work addresses the challenge of reproducing academic papers through automated code generation, which is often hindered by the absence of tacit knowledge—such as implementation nuances and debugging insights. The study presents the first systematic formalization of three types of tacit knowledge: relational, embodied, and collective. To recover this knowledge, the authors propose a graph-based agent framework that operates through a three-stage mechanism: relation-aware aggregation, execution-feedback refinement, and graph-level knowledge induction. They also introduce an expanded version of ReplicateBench, a large-scale evaluation benchmark encompassing three domains, ten tasks, and forty papers. Experimental results demonstrate that the generated code achieves an average performance gap of only 10.04% compared to official implementations, representing a 24.68% improvement over the strongest baseline.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Knowledge Representation and Reasoning: Knowledge Acquisition
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Automated paper reproduction -- generating executable code from academic papers -- is bottlenecked not by information retrieval but by the tacit knowledge that papers inevitably leave implicit. We formalize this challenge as the progressive recovery of three types of tacit knowledge -- relational, somatic, and collective -- and propose \method, a graph-based agent framework with a dedicated mechanism for each: node-level relation-aware aggregation recovers relational knowledge by analyzing implementation-unit-level reuse and adaptation relationships between the target paper and its citation neighbors; execution-feedback refinement recovers somatic knowledge through iterative debugging driven by runtime signals; and graph-level knowledge induction distills collective knowledge from clusters of papers sharing similar implementations. On an extended ReproduceBench spanning 3 domains, 10 tasks, and 40 recent papers, \method{} achieves an average performance gap of 10.04\% against official implementations, improving over the strongest baseline by 24.68\%. The code will be publicly released upon acceptance; the repository link will be provided in the final version.

Problem

• tacit knowledge
• automated paper reproduction
• relational knowledge
• somatic knowledge
• collective knowledge

Innovation

• tacit knowledge recovery
• graph-based agent framework
• automated paper reproduction
• execution-feedback refinement
• knowledge induction