Predicting the Understandability of Computational Notebooks through Code Metrics Analysis

2024-06-16

Empirical Software Engineering

AI Summary

To address the challenge of accurately assessing code understandability in computational notebooks, this paper introduces UOCU (User Opinion Code Understandability), the first quantitative metric grounded in user comments and integrating semantic analysis of reviews, user interaction patterns, and browsing behavior. Complementing this, we develop a static-code-only understandability prediction model—marking the first high-accuracy modeling of notebook understandability using solely code-based features. Methodologically, we fine-tune DistilBERT to extract comment semantics and engineer 34 fine-grained static code metrics, then employ a Random Forest classifier for ordinal (three-level) understandability prediction. Evaluated on 132,000 Jupyter notebooks, UOCU significantly outperforms conventional metadata- or survey-based approaches, achieving 89% accuracy in three-class classification. This work establishes a novel, automated, and scalable paradigm for code quality assessment in computational notebooks.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Computational notebooks have become the primary coding environment for data scientists. However, research on their code quality is still emerging, and the code shared is often of poor quality. Given the importance of maintenance and reusability, understanding the metrics that affect notebook code comprehensibility is crucial. Code understandability, a qualitative variable, is closely tied to user opinions. Traditional approaches to measuring it either use limited questionnaires to review a few code pieces or rely on metadata such as likes and votes in software repositories. Our approach enhances the measurement of Jupyter notebook understandability by leveraging user comments related to code understandability. As a case study, we used 542,051 Kaggle Jupyter notebooks from our previous research, named DistilKaggle. We employed a fine-tuned DistilBERT transformer to identify user comments associated with code understandability. We established a criterion called User Opinion Code Understandability (UOCU), which considers the number of relevant comments, upvotes on those comments, total notebook views, and total notebook upvotes. UOCU proved to be more effective than previous methods. Furthermore, we trained machine learning models to predict notebook code understandability based solely on their metrics. We collected 34 metrics for 132,723 final notebooks as features in our dataset, using UOCU as the label. Our predictive model, using the Random Forest classifier, achieved 89% accuracy in predicting the understandability levels of computational notebooks.

Problem

• Assessing Jupyter notebook code understandability using user opinions
• Developing a hybrid metric combining comments, upvotes, and views
• Predicting notebook understandability with machine learning models

Innovation

• Leverages user opinions from software repositories
• Introduces User Opinion Code Understandability metric
• Trains Random Forest for 89% accuracy prediction