Genomic data processing with GenomeFlow

2025-03-19

AI Summary

To address the high computational and storage costs, as well as architectural complexity, arising from heterogeneous tool orchestration in large-scale genomic/transcriptomic analysis on cloud platforms, this paper proposes an automated cloud architecture generation and resource optimization framework. Built on Google Cloud Platform, it integrates workflow orchestration, dynamic resource scheduling, cost-aware task allocation, and containerized integration of heterogeneous bioinformatics tools, enabling high-concurrency, elastic scaling—from single machines to thousands of CPU cores—for datasets ranging from hundreds to tens of thousands of samples. Its key innovation is the first cost-driven, end-to-end automatic cloud architecture generation mechanism, significantly lowering the barrier to cloud adoption for bioinformaticians. Experiments across multiple international consortium datasets demonstrate average reductions of 40–60% in both analysis runtime and computational cost.

Technology Category

• Constraint Satisfaction and Optimization: Solvers and Tools
• Machine Learning: Hardware-aware ML
• Search and Optimization: Evolutionary Computation

Application Category

• Economics, Online Markets and Human Computation: Economic aspects and information design of Web, GenAI, and cloud computing
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Advances in genome sequencing technologies generate massive amounts of sequence data that are increasingly analyzed and shared through public repositories. On-demand infrastructure services on cloud computing platforms enable the processing of such large-scale genomic sequence data in distributed processing environments with a significant reduction in analysis time. However, parallel processing on cloud computing platforms presents many challenges to researchers, even skillful bioinformaticians. In particular, it is difficult to design a computing architecture optimized to reduce the cost of computing and disk storage as genomic data analysis pipelines often employ many heterogeneous tools with different resource requirements. To address these issues, we developed GenomeFlow, a tool for automated development of computing architecture and resource optimization on Google Cloud Platform, which allows users to process a large number of samples at minimal cost. We outline multiple use cases of GenomeFlow demonstrating its utility to significantly reduce computing time and cost associated with analyzing genomic and transcriptomic data from hundreds to tens of thousands of samples from several consortia. Here, we describe a step-by-step protocol on how to use GenomeFlow for a common genomic data processing task. We introduce this example protocol geared toward a bioinformatician with little experience in cloud computing.

Problem

• Optimize computing architecture for genomic data processing
• Reduce cost and time in large-scale genomic data analysis
• Simplify cloud computing for bioinformaticians with GenomeFlow

Innovation

• Automated computing architecture development on Google Cloud
• Resource optimization for genomic data processing
• Reduces computing time and cost significantly