presto

Using Presto (Trino) to run distributed SQL queries across data lakes and federated sources by configuring connectors (Hive, S3, MySQL), writing optimized SQL, tuning resource pools and memory settings, and integrating with BI tools for interactive analytics.

presto

12-Month Skill Trend

Momentum and market value over time
Trending
Score
+20 in 12 mo
96
12 mo agoNow
Career
Value
+$12K in 12 mo
$42K/year
12 mo agoNow

Recommended Survey Paper

Quick overview of the field
View more

Must-Read Papers

Most classic and influential ideas
View more

This work addresses the CPU computational bottleneck encountered by Presto in large-scale analytical queries by introducing the first deep integration of GPU execution into the Presto framework. The authors propose an end-to-end query processing architecture that retains data in GPU memory throughout the execution pipeline. By redesigning the execution engine, they enable GPU-aware scheduling, efficient data transfer from storage to GPU, and intra-operator data exchange directly within GPU memory in distributed environments. The system is built upon NVIDIA’s cuDF C++ library and guided by TPC-H benchmarks for optimization. Experimental results demonstrate that, under standard analytical workloads, the proposed approach achieves up to a 6× improvement in cost-performance ratio compared to native CPU-based Presto. The implementation has been upstreamed into the open-source Presto/Velox engine and deployed in production.

data movementdistributed queryGPU acceleration

Rethinking Analytical Processing in the GPU Era

Aug 06, 2025
BY
Bobbi Yogatama
🏛️ University of Wisconsin-Madison | OrionX | NVIDIA | Posit PBC | Oracle

To address the limited adoption of GPU-accelerated data analytics due to insufficient hardware-software co-design, this paper proposes Sirius—a native GPU-accelerated SQL engine that fundamentally rearchitects analytical database systems with GPU as the primary execution engine. Methodologically, Sirius implements GPU-native relational operators atop libcudf and leverages the Substrait standard for portable, cross-system query plan exchange, enabling decoupled, plug-and-play integration with systems such as DuckDB and Apache Doris. Its key contributions are: (1) eliminating CPU-GPU hybrid execution in favor of end-to-end GPU-native processing; and (2) establishing a standardized, portable heterogeneous acceleration interface via Substrait. Experiments demonstrate up to 7× speedup for single-node DuckDB integration and 12.5× peak throughput improvement for distributed Doris integration—both at equivalent hardware cost—significantly advancing the practical deployment of GPUs in interactive analytical workloads.

Drop-in acceleration for existing database systemsGPU-native SQL engine for faster data analyticsOvercoming hardware and software barriers in GPU adoption

Data Management System Analysis for Distributed Computing Workloads

Oct 01, 2025
KH
Kuan-Chieh Hsu
🏛️ Brookhaven National Laboratory | University of Pittsburgh | Carnegie Mellon University | Oak Ridge National Laboratory | University of Massachusetts | SLAC National Accelerator Laboratory

To address systemic inefficiencies—including resource idleness, redundant data transfers, and violations of data locality—arising from misaligned coordination between the PanDA workflow system and the Rucio data management system in the ATLAS experiment, this paper proposes an end-to-end co-optimization framework. We introduce a novel file-level metadata matching algorithm to precisely associate computing tasks with datasets, and integrate log-based tracing, spatiotemporal imbalance analysis, and anomaly pattern detection to construct a fine-grained, holistic view of data access and movement. Our approach is the first to identify, in production, the root causes of cross-system scheduling mismatches, delivering interpretable performance insights. Empirical validation confirms tangible improvements in resource utilization and system resilience, demonstrating the feasibility and effectiveness of the proposed co-design strategies.

Diagnosing systemic inefficiencies in globally distributed data workflowsLinking workflow and data management systems for performance awarenessReducing unnecessary data transfers through coordinated system optimization

This work addresses the challenges of traditional AI analytics in multi-tenant heterogeneous data systems, including high computational overhead, limited robustness to data drift, and heightened security risks. To overcome these limitations, the paper proposes a deep integration of AI capabilities into the database engine, co-optimizing query processing and model execution while re-engineering transaction management and access control mechanisms to support efficient end-to-end AI lifecycle operations. The authors design a unified optimization and secure execution framework for AI-native databases (AIxDB), which synergistically combines query optimization, distributed scheduling, heterogeneous hardware adaptation, and fine-grained security enforcement. Preliminary experiments demonstrate the architecture’s effectiveness in mitigating data drift and optimizing resource scheduling, offering a foundational technical pathway toward secure, high-performance AI-native database systems.

access controlAIxDBdata drift

This work addresses the limitations of traditional workflow platforms, which rely on static, pre-defined processes and struggle to accommodate the dynamic data integration demands of distributed systems. To overcome this, the authors propose a configuration-driven runtime orchestration framework that dynamically constructs execution graphs at request time through dependency-aware scheduling and parallel task execution, thereby circumventing the constraints of fixed workflows. This approach enables rapid adaptation to evolving integration scenarios without requiring system redeployment, significantly reducing latency. Empirical evaluation in a real-world Customer 360 enterprise use case demonstrates that the framework offers substantial advantages in flexibility, scalability, and efficient data aggregation compared to conventional solutions.

distributed systemsdynamic data retrievalheterogeneous integrations

Latest Papers

What's happening recently
View more

This work addresses the scarcity of real-world SQL workloads and limitations of existing synthetic query generation methods—such as rigid templates, schema hallucination, and simplistic join structures—that hinder effective training of query optimizers. The authors propose SynQL, a rule-based, deterministic SQL synthesis framework that constructs abstract syntax trees by traversing foreign key graphs in database schemas, eschewing probabilistic generation to guarantee syntactic and schema-level validity. SynQL introduces a configuration vector to explicitly control join topology, analytical complexity, and predicate selectivity, while modeling core query components including multi-table joins, projections, aggregations, and range predicates. Workloads generated on TPC-H and IMDb achieve a topological entropy of 1.53 bits; tree-based models trained on this data attain R² ≥ 0.79 on test sets with sub-millisecond inference latency.

query optimisationschema hallucinationSQL workload synthesis

This work addresses the limitations of existing SQL rewriting approaches, which either rely on rigid rule-based systems lacking adaptability or depend on large language models (LLMs) that incur high computational costs and privacy concerns. To overcome the scarcity of high-quality domain-specific data for training small language models, the authors propose LASER, a novel framework that integrates Monte Carlo Tree Search (MCTS) with LLM-guided mutation to generate SQL-MCTS—a large-scale corpus of complex, slow queries. They further introduce SQL-GRPO, a group-relative policy optimization algorithm featuring anchored group advantages and a complexity-adaptive dynamic rollout mechanism. Experiments on the Qwen3 small model demonstrate that LASER achieves an exceptional trade-off among execution efficiency, zero-shot transfer capability, and inference overhead, significantly outperforming both rule-based systems and LLM-based alternatives.

database efficiencyquery optimizationSmall Language Models

Multi-cloud ETL faces significant challenges, including high cross-cloud data movement costs, incompatibility among heterogeneous SQL engines, complex orchestration, and fragmented security policies. This work presents the first systematic evaluation of predicate pushdown optimization in multi-cloud ETL scenarios and proposes a novel协同 strategy that integrates localized pushdown with data federation to offload transformation logic across heterogeneous query engines such as Amazon Redshift and Google BigQuery. By intelligently pushing computation closer to the data sources while leveraging federated querying capabilities, the approach substantially reduces cross-cloud data transfer, thereby lowering end-to-end execution latency and cost. The proposed paradigm offers an efficient and scalable solution for optimizing multi-cloud data integration workflows.

data integrationETLheterogeneous databases

In data lake queries, decoding Parquet files from remote storage can account for up to 46% of runtime overhead. To address this, this work proposes a cloud-oriented SmartNIC architecture that, for the first time, leverages SmartNICs to directly process pre-filtered columnar data. By offloading Parquet decoding and pushing down query operators onto the network data path, the design enables hardware-accelerated query execution. Integrated with DuckDB, the approach substantially reduces CPU resource consumption. Experimental results demonstrate that after SmartNIC-based pre-filtering, a modest CPU configuration achieves query throughput comparable to that of conventional systems relying on significantly more powerful host processors.

data lakedecoding overheadperformance bottleneck

This work addresses the tendency of existing large language model (LLM)-driven Text-to-SQL agents to over-explore in fine-grained API environments, often incorporating irrelevant schema elements and thereby degrading SQL generation accuracy. To mitigate this issue, the paper introduces a novel mechanism that embeds guiding instructions directly within data system API responses to actively regulate the agent’s exploration behavior. This approach enhances exploration efficiency while preserving query precision. Notably, it represents the first method to leverage API response instructions for constraining excessive exploration. Experimental results demonstrate that the proposed guiding instructions reduce over-exploration by a factor of 4.6 and improve SQL generation accuracy by up to 12.4% (approximately 4 percentage points).

API granularityLLM agentsover-exploration

Hot Scholars

QW

Qiuyue Wang

School of Information, Renmin University of China
information extractionknowledge graphknowledge reasoning
XY

Xin Yan

Missouri University of S&T, Google
DM

Deepak Majeti

Ahana
Big Data ArchitecturesParallel Programming LanguagesCompilersParallel Runtime