Query Optimization in the Wild: Realities and Trends

2025-10-22

AI Summary

Traditional query optimizers—based on System R, Volcano, or Cascades—employ a monolithic, static, single-query architecture that exhibits performance instability, lacks global workload-level optimization, and suffers from architectural rigidity in cloud-native environments with massive-scale data and unified data platforms. To address these industrial challenges, this paper proposes three evolutionary directions: (1) extending optimization from individual queries to workload-aware collaborative optimization; (2) establishing an execution-feedback-driven closed-loop optimization framework; and (3) designing a composable, modular optimizer architecture enabling cross-engine reuse and agile iteration. The study distills three key trends—workload awareness, feedback closure, and architectural decoupling—providing a practical, implementable technical pathway and industrial paradigm for building next-generation query optimization systems that are dynamic, holistic, and self-adaptive.

Technology Category

• Search and Optimization: Evolutionary Computation
• Machine Learning: Optimization
• Constraint Satisfaction and Optimization: Constraint Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

For nearly half a century, the core design of query optimizers in industrial database systems has remained remarkably stable, relying on foundational principles from System R and the Volcano/Cascades framework. However, the rise of cloud computing, massive data volumes, and unified data platforms has exposed the limitations of this traditional, monolithic architecture. Taking an industrial perspective, this paper reviews the past and present of query optimization in production systems and identifies the challenges they face today. Then this paper highlights three key trends gaining momentum in the industry that promise to address these challenges. First, a tighter feedback loop between query optimization and query execution is being used to improve the robustness of query performance. Second, the scope of optimization is expanding from a single query to entire workloads through the convergence of query optimization and workload optimization. Third, and perhaps most transformatively, the industry is moving from monolithic designs to composable architectures that foster agility and cross-engine collaboration. Together, these trends chart a clear path toward a more dynamic, holistic, and adaptable future for query optimization in practice.

Problem

• Addressing limitations of traditional monolithic query optimizer architecture
• Improving query performance robustness through optimization-execution feedback loops
• Expanding optimization scope from single queries to entire workloads

Innovation

• Feedback loop between optimization and execution improves robustness
• Expanding optimization scope from single queries to workloads
• Moving from monolithic designs to composable architectures