Are the Majority of Public Computational Notebooks Pathologically Non-Executable?

2025-02-06

AI Summary

This paper challenges the overemphasis on “non-executability” in public computational notebooks, arguing that the conventional requirement of error-free execution across all cells is overly stringent and neglects the interactive nature of notebook environments. Method: We propose a progressive executability assessment framework that formally defines and quantifies “recoverably non-executable” notebooks—distinguishing repairable cases from truly pathological ones. Our approach integrates LLM-driven code repair, automated module dependency identification and installation, synthetic data generation, and multi-level executability metrics. Results: Evaluated on 42,000 real-world notebooks, our method fully recovers 5.4% of non-executable notebooks via LLM repair; dependency resolution and synthetic data improve cell-level executability by 42.7% and 28%, respectively; ultimately, only 21.3% are confirmed as irrecoverably pathological—correcting 76% of prior false non-executability judgments. The core contribution is a paradigm shift in how executability is conceptualized, substantially enhancing notebook reusability and code understanding accuracy.

Technology Category

• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Probabilistic Programming
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Computational notebooks are the de facto platforms for exploratory data science, offering an interactive programming environment where users can create, modify, and execute code cells in any sequence. However, this flexibility often introduces code quality issues, with prior studies showing that approximately 76% of public notebooks are non-executable, raising significant concerns about reusability. We argue that the traditional notion of executability - requiring a notebook to run fully and without error - is overly rigid, misclassifying many notebooks and overestimating their non-executability. This paper investigates pathological executability issues in public notebooks under varying notions and degrees of executability. Even partially improving executability can improve code comprehension and offer a pathway for dynamic analyses. With this insight, we first categorize notebooks into potentially restorable and pathological non-executable notebooks and then measure how removing misconfiguration and superficial execution issues in notebooks can improve their executability (i.e., additional cells executed without error). In a dataset of 42,546 popular public notebooks containing 34,659 non-executable notebooks, only 21.3% are truly pathologically non-executable. For restorable notebooks, LLM-based methods fully restore 5.4% of previously non-executable notebooks. Among the partially restored, the notebook extquotesingle s executability improves by 42.7% and 28% by installing the correct modules and generating synthetic data. These findings challenge prior assumptions, suggesting that notebooks have higher executability than previously reported, many of which offer valuable partial execution, and that their executability should be evaluated within the interactive notebook paradigm rather than through traditional software executability standards.

Problem

• Assessing executability of public computational notebooks.
• Challenging rigid traditional executability standards for notebooks.
• Improving notebook executability via LLM-based restoration methods.

Innovation

• Categorizes notebooks into restorable and pathological
• Uses LLM-based methods to restore executability
• Improves executability via module installation and synthetic data