Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraft

2026-04-27

AI Summary

Current AI systems face a significant complexity gap between scientific tasks like causal discovery and real-world engineering applications. This work proposes SciCrafter—the first evaluable benchmark based on Minecraft redstone circuits—that operationalizes the discovery-to-application loop into a measurable challenge through a parameterized “light bulbs in specified patterns” task. The authors introduce a four-stage capability decomposition framework to systematically assess model performance. Experimental results reveal that state-of-the-art models, including GPT-5.2 and Gemini-3-Pro, achieve only a 26% average success rate, indicating that the primary bottleneck has shifted from solving problems to formulating the right questions. Crucially, deficiencies in knowledge application and problem identification emerge as key limitations hindering effective deployment in such closed-loop scientific-engineering scenarios.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Machine Learning: Probabilistic Circuits and Graphical Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Discovering causal regularities and applying them to build functional systems--the discovery-to-application loop--is a hallmark of general intelligence, yet evaluating this capacity has been hindered by the vast complexity gap between scientific discovery and real-world engineering. We introduce SciCrafter, a Minecraft-based benchmark that operationalizes this loop through parameterized redstone circuit tasks. Agents must ignite lamps in specified patterns (e.g., simultaneously or in timed sequences); scaling target parameters substantially increases construction complexity and required knowledge, forcing genuine discovery rather than reliance on memorized solutions. Evaluating frontier models including GPT-5.2, Gemini-3-Pro, and Claude-Opus-4.5 under a general-purpose code agent scaffold, we find that all plateau at approximately 26% success rate. To diagnose these failures, we decompose the loop into four capacities--knowledge gap identification, experimental discovery, knowledge consolidation, and knowledge application--and design targeted interventions whose marginal contributions serve as proxies for corresponding gaps. Our analysis reveals that although the general knowledge application capability still remains as the biggest gap across all models, for frontier models the knowledge gap identification starts to become a major hurdle--indicating the bottleneck is shifting from solving problems right to raising the right problems for current AI. We release SciCrafter as a diagnostic probe for future research on AI systems that navigate the full discovery-to-application loop.

Problem

• discovery-to-application gap
• causal regularities
• knowledge gap identification
• general intelligence
• AI agent evaluation

Innovation

• discovery-to-application loop
• SciCrafter
• causal discovery
• knowledge gap identification
• Minecraft benchmark