snowflake

Data engineering and AI/ML platforms face inherent trade-offs among performance, security, usability, and seamless integration with existing data architectures. Method: This paper proposes Snowpark—a platform built upon Snowflake’s elastic architecture—that (1) introduces a Python package caching mechanism to drastically reduce query initialization latency; (2) implements a customized workload scheduler with row-level redistribution to mitigate data skew; and (3) enforces tenant-level isolation via secure sandboxes, enabling robust multi-language (especially Python) support and deep control-plane integration. Results: Evaluated in production environments, the solution improves execution efficiency by 37%, increases resource utilization by 2.1×, and ensures strict tenant isolation and zero-trust security—establishing a scalable, secure, low-latency systems paradigm for cloud-native data intelligence platforms.

Existing predicate-driven partition pruning techniques struggle to support critical operators such as LIMIT, top-k, and JOIN, severely limiting analytical efficiency over large-scale cloud data. This paper introduces the first partition pruning paradigm explicitly designed for LIMIT/top-k/JOIN workloads. Leveraging high-selectivity query characteristics, we propose a dynamic pruning algorithm that exploits min/max metadata to enable runtime, adaptive skipping of irrelevant partitions. Our approach is deeply integrated with modern data lake formats—including Apache Iceberg—and co-optimized with Snowflake’s micro-partition execution engine. Experiments on Snowflake’s production environment demonstrate a 99.4% reduction in micro-partitions processed, yielding substantial improvements in throughput and latency for complex analytical queries. This work advances benchmark design toward realism, significantly extending both the technical boundaries and practical applicability of partition pruning.

This work addresses the susceptibility of Snowpark UDF execution to data skew induced by user-defined logic, which leads to task delays and inefficient resource utilization. To mitigate this issue, the authors propose a dynamic, fine-grained redistribution mechanism that integrates a state machine–based adaptive data distribution strategy, an eager redistribution policy, and a row-size prediction model to accurately detect and alleviate skew at runtime. Implemented within Snowflake’s general-purpose skew-handling framework, the approach leverages per-link state machines, dynamic row-level redistribution, and cost-aware scheduling to significantly reduce both execution time and resource consumption for large-scale UDF workloads. Experimental results demonstrate its superior performance over conventional static round-robin strategies.

Large language models (LLMs) exhibit significant capability gaps in real-world enterprise text-to-SQL tasks, where complexity arises from multi-query generation, cross-dialect SQL, long-context reasoning, and metadata-dependent schema retrieval. Method: We introduce the first enterprise workflow-oriented benchmark—comprising 632 complex, real-world questions drawn from production BigQuery and Snowflake databases—and propose a novel code-intelligent agent framework based on o1-preview, supporting multi-step planning, execution verification, and project-level code-context retrieval. Contribution/Results: We systematically formalize full-stack enterprise SQL challenges, incorporating new evaluation dimensions: hundred-line multi-operation SQL generation, joint database schema and documentation augmentation, and cross-dialect compatibility. State-of-the-art models achieve only 17.0% exact match accuracy—far below Spider (91.2%) and BIRD (73.0%)—highlighting critical deficiencies in LLMs’ deep structured-data understanding and collaborative reasoning under industrial constraints.

Open-source large language models (LLMs) exhibit weak generalization and severe SQL hallucination in Text-to-SQL tasks. Method: We propose a robust multi-task fine-tuning and collaborative reasoning framework. Specifically, we introduce the first multi-task supervised fine-tuning (SFT) that jointly models schema linking, noise correction, and SQL continuation. We further design Multi-Task Collaborative Prompting (MCP) to explicitly capture inter-task dependencies and suppress hallucination. Combined with synthetic data augmentation and open-LLM adaptation techniques, our approach enhances robustness against complex schemas and noisy inputs. Contribution/Results: Extensive experiments across eight mainstream open-source LLMs and five benchmarks—including Spider and BIRD—demonstrate consistent and significant improvements over prior methods, achieving state-of-the-art performance in the Text-to-SQL domain. Our framework delivers a practical, reliable solution for open-scenario Text-to-SQL applications.

This work addresses the unreliability of developer productivity dashboards, which often stems from ad hoc scripts that introduce undetected silent data gaps, eroding organizational trust. To resolve this, we propose a robust ELT pipeline grounded in DAG-based orchestration and the Medallion architecture, decoupling data extraction from transformation to preserve the immutability of raw data. Our approach introduces a state-driven dependency scheduling mechanism and, for the first time, treats metric pipelines as production-grade distributed systems. We emphasize the critical role of immutable raw history in enabling reliable metric redefinition. This methodology significantly enhances data reliability and freshness while effectively eliminating silent failures, thereby restoring organizational confidence in DevOps metrics.

This work addresses the challenges of natural language to SQL (NL2SQL) translation in real-world enterprise databases, where complex table schemas, opaque column names, dialect heterogeneity, and deeply nested queries hinder performance. To tackle these issues, the authors propose a semantic-layer mediation mechanism that introduces Semantic Model Queries (SMQ) as an intermediate representation, decoupling user intent from physical SQL generation. They further design a constrained think-execute loop and a deterministic compiler to prevent overfitting to the raw database schema. Built upon the Gemini 3 Pro large language model and supporting SQLite, BigQuery, and Snowflake backends, the system achieves a 94.15% execution accuracy on the 547 tasks of Spider2-snow, ranking third on the official leaderboard and significantly outperforming approaches that rely solely on the original schema, thereby substantially enhancing cross-dialect NL2SQL generalization.

This work addresses the absence of Git-like data versioning in existing lakehouse architectures, which hinders collaborative development among multiple agents and human-in-the-loop review workflows. To bridge this gap, the paper introduces Git semantics into lakehouse systems by extending Apache Iceberg to support lakehouse-wide commits, branches, and merges—elevating single-table snapshots to a unified versioning model. This enables agents to operate on isolated branches while ensuring atomic, cross-table changes during release. The core abstractions are formally modeled and verified using Alloy to guarantee semantic correctness. The proposed system has been deployed in production, demonstrating the feasibility and effectiveness of the collaborative workflow it enables.

Existing text-to-SQL approaches struggle with enterprise databases due to schema heterogeneity, missing metadata, diverse SQL dialects, and complex analytical tasks. To address these challenges, this work proposes ProSPy, a novel framework that introduces data-probing-driven proxy reasoning into the domain for the first time. ProSPy automatically extracts fine-grained evidence through data probing, progressively prunes the schema to retain only task-relevant elements, generates dialect-agnostic intermediate views via a unified SQL interface, and leverages Python for flexible downstream analysis. This approach substantially reduces reliance on metadata and enhances robustness across SQL dialects, achieving execution accuracies of 60.15% and 60.51% on Spider 2.0-Lite and Spider 2.0-Snow, respectively—significantly outperforming strong baselines.

This work addresses the tendency of existing large language model (LLM)-driven Text-to-SQL agents to over-explore in fine-grained API environments, often incorporating irrelevant schema elements and thereby degrading SQL generation accuracy. To mitigate this issue, the paper introduces a novel mechanism that embeds guiding instructions directly within data system API responses to actively regulate the agent’s exploration behavior. This approach enhances exploration efficiency while preserving query precision. Notably, it represents the first method to leverage API response instructions for constraining excessive exploration. Experimental results demonstrate that the proposed guiding instructions reduce over-exploration by a factor of 4.6 and improve SQL generation accuracy by up to 12.4% (approximately 4 percentage points).