pandas

This work addresses the significant limitations of spreadsheet-based analysis in reproducibility, auditability, version control, and automation. It proposes a migration pathway from Excel to research-grade analytical workflows by leveraging Python’s pandas library as a bridge. The study introduces an innovative set of Excel-to-pandas mapping rules, categorizes nine canonical workflow patterns, and compiles a catalog of common failure modes. Seven end-to-end real-world examples demonstrate the approach in practice. By retaining Excel as a familiar interface for input and output while integrating version control, automated refreshing, and seamless incorporation of statistical and machine learning methods, the proposed framework enables governed, reproducible, and auditable tabular data analysis.

Pandas struggles with ultra-large-scale data due to memory constraints, and migrating to scalable frameworks (e.g., Dask, Modin) requires extensive code rewriting. Method: This paper introduces the “Lazy Fat Panda Diet” paradigm—a JIT-based static analysis and lazy-loading DataFrame wrapper that enables seamless, API-compatible, backend-agnostic optimization across Pandas, Dask, and Modin—without modifying user code. The system supports lazy execution and runtime memory-aware query planning via just two lines of code. Contribution/Results: Evaluated on real-world data science workloads, our approach achieves 2.1–5.8× end-to-end speedup over baseline Pandas, maintains correctness and stability even under memory pressure, and eliminates costly framework migration. It significantly reduces development and deployment overhead for large-scale analytics while preserving full Pandas compatibility.

This work addresses the lack of standardized evaluation for large language models (LLMs) in generating visualization code from tabular data. We introduce PandasPlotBench, the first benchmark dataset specifically targeting Pandas DataFrame visualization—comprising 175 human-annotated tasks spanning Matplotlib, Seaborn, and Plotly. Our methodology employs natural-language instruction-driven code generation, coupled with standardized prompting and execution-based code evaluation. Key contributions include: (1) the first fine-grained, task-level evaluation framework for exploratory data visualization from tables; (2) empirical identification of a significant generalization gap for LLMs on Plotly—despite strong performance on Matplotlib and Seaborn; (3) demonstration of robustness to concise instructions across the latter two libraries; and (4) open-sourcing of both the benchmark dataset and evaluation framework, which has been widely adopted by the research community.

Heterogeneous tabular data pose significant integration challenges due to inconsistencies in format and semantics. Method: This paper proposes the first large language model (LLM)-based framework for multi-category table transformation. It introduces a novel automatic classification mechanism for four transformation types—string, numeric, algorithmic, and generic—and leverages table semantic understanding, rule-guided prompt engineering, multi-stage task decomposition, and function synthesis to generate human-readable, editable, function-level transformations (e.g., Excel formulas or Python code). Contribution/Results: Evaluated on real-world datasets across multiple domains, the framework achieves statistically significant accuracy improvements over state-of-the-art methods, supports a broader spectrum of transformation types, and ensures full human interpretability of all outputs—thereby balancing automation capability with transparency and explainability.

Existing Jupyter-style notebooks employ a linear execution model, which poorly supports data scientists’ typical nonlinear exploratory behaviors—such as backward execution, branch switching, and full-state rollback. To address this, we propose Kishuboard: a novel computational notebook system built upon two-dimensional versioning of both code and data spaces. Kishuboard introduces joint code-and-data state snapshots, constructs a dependency-aware version graph, and designs an overlayable one-dimensional history view to balance intuitiveness and expressive power. Interactive navigation controls enable flexible multi-branch state management. A user study demonstrates that Kishuboard significantly improves efficiency and task completion rates for complex exploratory workflows. By unifying and scaling versioning across code and data states, Kishuboard provides principled, extensible support for nonlinear data science workflows.

Prior work lacks systematic evaluation of how data processing frameworks impact end-to-end deep learning training and inference—particularly regarding performance-energy trade-offs across data loading, preprocessing, and batch feeding stages in conjunction with GPU computation. Method: We conduct the first comprehensive empirical study comparing Pandas, Polars, and Dask across diverse deep learning workloads—including CNNs and Transformers trained on ImageNet and WikiText—measuring runtime, memory footprint, disk I/O, and CPU/GPU power consumption under varying data scales and I/O characteristics. Contribution/Results: Polars achieves optimal latency–energy efficiency for medium-scale in-memory datasets; Dask scales effectively to ultra-large distributed workloads but exhibits lower energy efficiency; Pandas remains practical for small-batch, interactive tasks. Our findings bridge a critical gap in co-optimizing data engineering infrastructure with AI training pipelines, providing empirical guidance for green AI system design and framework selection in production ML systems.

This study systematically evaluates performance differences among Java, Python, and Scala for end-to-end ETL workloads on Apache Spark integrated with Apache Iceberg. We conduct controlled experiments across varying data scales (5 MB–1.6 GB) and operation complexities (basic transformations vs. complex merge operations), all within a uniform CSV→Spark→Iceberg pipeline. To our knowledge, this is the first standardized ETL benchmark enabling direct cross-language comparison under native Iceberg support. Results reveal nonlinear interactions among programming language, data volume, and operation type: Python exhibits superior throughput on small-scale data; performance converges across all three languages at medium scale (1.6 GB); and Scala significantly outperforms both Java and Python in high-complexity merge-intensive workloads. The findings provide empirical guidance for language selection in big-data ETL systems, along with a principled trade-off framework grounded in workload characteristics.

This paper addresses the lack of a unified, semantically clear syntactic framework for data analysis. It proposes a novel, operation-agnostic analytical syntax grounded in two primitive constructs—“measures” and “dimensions”—thereby abstracting analytical logic and decoupling semantic intent from underlying data sources. This design significantly enhances readability, reusability, and maintainability of analytical expressions. To our knowledge, this is the first systematic formulation of such a syntax model. Based on it, we design and open-source Meterstick, a Python library supporting unified analysis across heterogeneous backends—including pandas DataFrames and SQL databases. Empirical evaluation demonstrates that the framework delivers efficient execution while enabling cross-platform, declarative, and composable analytical modeling. By providing a foundational syntax layer, it advances the Analytics-as-Code paradigm.

We address the problem of evaluating conjunctive queries with degree constraints and disjunctive Datalog rules (DDRs). We propose the first algorithmic framework that is both fully general—supporting arbitrary degree bounds, free variables, and Boolean/non-Boolean queries—and asymptotically optimal in time complexity. To eliminate the dominant polylogarithmic overhead inherent in PANDA, we introduce a data-skew-aware dynamic hyperplane partitioning scheme and derive a novel probabilistic inequality. Integrating submodular width theory with explicit degree-constraint modeling, our algorithm achieves the tight complexity bound $ ilde{O}(N^{ ext{subw}})$, matching the precision of specialized algorithms while natively supporting common integrity constraints such as functional dependencies. Experimental results demonstrate substantial performance gains over PANDA, alongside greater simplicity and practical applicability.

This work addresses the challenge of synchronously comparing event sequences with heterogeneous tabular attributes—such as numerical, categorical, and temporal data—in visual analytics. To this end, the authors propose EventColumn, a novel column type that unifies event sequences and diverse attribute types within a single coordinated view, enabling both instance-level and cohort-level joint analysis. Through interactive mechanisms including compact overviews, heatmap-based summaries, event alignment, and boxplots of historically similar items, the approach significantly enhances analytical efficiency. Implemented in Taggle and Microsoft Power BI, the method has been validated on real-world datasets from steel production logistics and public e-commerce domains, demonstrating its effectiveness in structured event data scenarios and substantially improving users’ ability to discern relationships between events and associated attributes.