๐ค AI Summary
Multi-cloud ETL faces significant challenges, including high cross-cloud data movement costs, incompatibility among heterogeneous SQL engines, complex orchestration, and fragmented security policies. This work presents the first systematic evaluation of predicate pushdown optimization in multi-cloud ETL scenarios and proposes a novelๅๅ strategy that integrates localized pushdown with data federation to offload transformation logic across heterogeneous query engines such as Amazon Redshift and Google BigQuery. By intelligently pushing computation closer to the data sources while leveraging federated querying capabilities, the approach substantially reduces cross-cloud data transfer, thereby lowering end-to-end execution latency and cost. The proposed paradigm offers an efficient and scalable solution for optimizing multi-cloud data integration workflows.
๐ Abstract
Enterprises increasingly adopt multi cloud architectures to take advantage of diverse database engines, regional availability, and cost models. In these environments, ETL pipelines must process large, distributed datasets while minimizing latency and transfer cost. Push down optimization, which executes transformation logic within database engines rather than within the ETL tool, has proven highly effective in single cloud systems. However, when applied across multiple clouds, it faces challenges related to data movement, heterogeneous SQL engines, orchestration complexity, and fragmented security controls. This paper examines the feasibility of push down optimization in multi cloud ETL pipelines and analyzes its benefits and limitations. It evaluates localized push down, hybrid models, and data federation techniques that reduce cross cloud traffic while improving performance. A case study across Redshift and BigQuery demonstrates measurable gains, including lower end to end runtime, reduced transfer volume, and improved cost efficiency. The study highlights practical strategies that organizations can adopt to improve ETL scalability and reliability in distributed cloud environments.