python data analysis & visualization

This work addresses the lack of standardized evaluation for large language models (LLMs) in generating visualization code from tabular data. We introduce PandasPlotBench, the first benchmark dataset specifically targeting Pandas DataFrame visualization—comprising 175 human-annotated tasks spanning Matplotlib, Seaborn, and Plotly. Our methodology employs natural-language instruction-driven code generation, coupled with standardized prompting and execution-based code evaluation. Key contributions include: (1) the first fine-grained, task-level evaluation framework for exploratory data visualization from tables; (2) empirical identification of a significant generalization gap for LLMs on Plotly—despite strong performance on Matplotlib and Seaborn; (3) demonstration of robustness to concise instructions across the latter two libraries; and (4) open-sourcing of both the benchmark dataset and evaluation framework, which has been widely adopted by the research community.

This work addresses the challenge of reproducibility in data visualization scripts, which often lack essential components such as source code, input data, execution environment, or output artifacts. To bridge this gap, we propose yProv4DV—a lightweight Python library that, for the first time, targets script-based visualization workflows by automatically capturing comprehensive provenance information—including source code, input data, runtime environment, and output results—through a single function call. Designed to be minimally invasive and ready-to-use, yProv4DV enables full reproducibility of visualization outputs without requiring any modification to existing scripts. This approach significantly reduces the development burden on researchers striving to ensure reproducibility and fills a critical void in automated provenance support within visualization pipelines.

This work addresses the significant limitations of spreadsheet-based analysis in reproducibility, auditability, version control, and automation. It proposes a migration pathway from Excel to research-grade analytical workflows by leveraging Python’s pandas library as a bridge. The study introduces an innovative set of Excel-to-pandas mapping rules, categorizes nine canonical workflow patterns, and compiles a catalog of common failure modes. Seven end-to-end real-world examples demonstrate the approach in practice. By retaining Excel as a familiar interface for input and output while integrating version control, automated refreshing, and seamless incorporation of statistical and machine learning methods, the proposed framework enables governed, reproducible, and auditable tabular data analysis.

Pandas struggles with ultra-large-scale data due to memory constraints, and migrating to scalable frameworks (e.g., Dask, Modin) requires extensive code rewriting. Method: This paper introduces the “Lazy Fat Panda Diet” paradigm—a JIT-based static analysis and lazy-loading DataFrame wrapper that enables seamless, API-compatible, backend-agnostic optimization across Pandas, Dask, and Modin—without modifying user code. The system supports lazy execution and runtime memory-aware query planning via just two lines of code. Contribution/Results: Evaluated on real-world data science workloads, our approach achieves 2.1–5.8× end-to-end speedup over baseline Pandas, maintains correctness and stability even under memory pressure, and eliminates costly framework migration. It significantly reduces development and deployment overhead for large-scale analytics while preserving full Pandas compatibility.

This study investigates common quality attributes characterizing high-impact Jupyter Notebooks. To address the lack of cross-platform empirical evidence on data science artifacts, we conduct the first comparative analysis of highly upvoted Kaggle notebooks and highly starred GitHub notebooks. Methodologically, we integrate exploratory data analysis, SHAP-based interpretability analysis, random forest feature importance assessment, and structural parsing of code and narrative text to systematically quantify three core dimensions: documentation completeness, modular design, and reproducibility configuration. Results show that these three features collectively account for over 75% of predictive importance for quality classification. Based on this, we propose an actionable Notebook quality assessment framework and a beginner-oriented practical guideline. This work establishes the first empirically grounded, cross-platform benchmark for evaluating Jupyter Notebooks—directly supporting pedagogy, peer review, and tool development in data science education and practice.

This work addresses the widespread misuse of statistical methods—often stemming from implicit or ambiguous assumptions—which exacerbates the reproducibility crisis in scientific research, particularly in hypothesis testing and meta-analysis where formal verification mechanisms are lacking. To bridge this gap, the authors propose the first formal verification framework tailored for Python-based statistical programs. By developing a Why3-py frontend, they translate dynamically typed, runtime-polymorphic Python code into the WhyML intermediate representation and extend the StatWhy tool to support meta-analysis verification. Integrating program transformation, static analysis, and formal verification techniques, this approach enables, for the first time, automated correctness verification of statistical programs written in Python, effectively uncovering overlooked assumptions and misuses, thereby filling a critical void in the formal verification of statistical software.

Prior work lacks systematic evaluation of how data processing frameworks impact end-to-end deep learning training and inference—particularly regarding performance-energy trade-offs across data loading, preprocessing, and batch feeding stages in conjunction with GPU computation. Method: We conduct the first comprehensive empirical study comparing Pandas, Polars, and Dask across diverse deep learning workloads—including CNNs and Transformers trained on ImageNet and WikiText—measuring runtime, memory footprint, disk I/O, and CPU/GPU power consumption under varying data scales and I/O characteristics. Contribution/Results: Polars achieves optimal latency–energy efficiency for medium-scale in-memory datasets; Dask scales effectively to ultra-large distributed workloads but exhibits lower energy efficiency; Pandas remains practical for small-batch, interactive tasks. Our findings bridge a critical gap in co-optimizing data engineering infrastructure with AI training pipelines, providing empirical guidance for green AI system design and framework selection in production ML systems.

To address the lack of early, machine-readable descriptions of scientific data analysis workflows—hindering FAIR (Findable, Accessible, Interoperable, Reusable) compliance—this paper introduces dtreg, the first structured registration framework for statistical and machine learning pipelines targeting the pre-publication stage and supporting both Python and R. Its core contributions are: (1) a novel pre-analysis metadata registration mechanism; (2) a persistent, globally identifiable schema system covering mainstream statistical tests (e.g., t-tests) and ML methods; and (3) lightweight, automated RDF/Linked Data serialization to Turtle and JSON-LD. Leveraging object-oriented modeling, dynamic schema population, and export capabilities, dtreg enables end-to-end machine-readable workflow documentation. As an open-source infrastructure, it significantly enhances the findability, interoperability, and reusability of analytical methods in computational research.

This study addresses the lack of systematic understanding regarding the effectiveness and usage practices of univariate distribution visualizations across diverse tasks and user groups. Through a mixed-methods approach—combining a click-based selection experiment and survey with 215 participants alongside in-depth interviews with five visualization practitioners—the work systematically evaluates the accuracy, user preferences, and common misinterpretations associated with boxplots, violin plots, jittered scatterplots, and histograms in typical analytical tasks. For the first time, it integrates task performance, subjective preference, and real-world practice, revealing a frequent mismatch between chart familiarity and task accuracy, thereby challenging the assumption that commonly used or conventional visualizations are inherently optimal. The findings demonstrate significant performance differences among chart types in low-level tasks, with widely adopted histograms and boxplots not consistently outperforming alternatives.

Traditional boxplots struggle to detect bimodal or multimodal structures in univariate data and fail to reveal underlying subgroups or outliers. To address this, we propose the *bixplot*, a novel boxplot variant that integrates a univariate clustering algorithm with continuity constraints and a minimum cluster size requirement to automatically identify and visualize meaningful subgroups within multimodal distributions. The bixplot supports direct plotting of individual data points and enables gradient color mapping based on external variables, thereby enhancing interpretability of substructures. Crucially, it guarantees non-nested clusters and ensures each cluster contains a sufficient number of distinct observations. Implemented as open-source Python and R packages, the bixplot demonstrates superior performance over conventional boxplots across multiple real-world datasets—clearly exposing previously overlooked multimodal patterns and substantially improving the depth and practical utility of exploratory data analysis.