Learned Query Optimizer in Alibaba MaxCompute: Challenges, Analysis, and Solutions

2026-02-07

AI Summary

This work addresses the challenges faced by existing learned query optimizers in dynamic, distributed multi-tenant data warehouses, where input statistics are often missing, online fine-tuning is impractical, and performance gains are uncertain. To overcome these limitations, the authors propose LOAM, a novel framework that introduces the first plan encoding method independent of input statistics, explicitly models the impact of execution environments on query cost, and integrates domain-adaptive training with a lightweight, high-yield item selection mechanism. This design enables efficient deployment without requiring online fine-tuning. Evaluated in Alibaba’s MaxCompute production environment, LOAM reduces CPU costs by up to 30% compared to the native optimizer, substantially lowering resource consumption.

Technology Category

• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Existing learned query optimizers remain ill-suited to modern distributed, multi-tenant data warehouses due to idealized modeling assumptions and design choices. Using Alibaba's MaxCompute as a representative, we surface four fundamental, system-agnostic challenges for any deployable learned query optimizer: 1) highly dynamic execution environments that induce large variance in plan costs; 2) potential absence of input statistics needed for cost estimation; 3) infeasibility of conventional model refinement; and 4) uncertain benefits across different workloads. These challenges expose a deep mismatch between theoretical advances and production realities and demand a principled, deployment-first redesign of learned optimizers. To bridge this gap, we present LOAM, a one-stop learned query optimization framework for MaxCompute. Its design principles and techniques generalize and are readily adaptable to similar systems. Architecturally, LOAM introduces a statistics-free plan encoding that leverages operator semantics and historical executions to infer details about data distributions and explicitly encodes the execution environments of training queries to learn their impacts on plan costs. For online queries with unknown environments at prediction time, LOAM provides a theoretical bound on the achievable performance and a practical strategy to smooth the environmental impacts on cost estimations. For system operating, LOAM integrates domain adaptation techniques into training to generalize effectively to online query plans without requiring conventional refinement. Additionally, LOAM includes a lightweight project selector to prioritize high-benefit deployment projects. LOAM has seen up to 30% CPU cost savings over MaxCompute's native query optimizer on production workloads, which could translate to substantial real-world resource savings.

Problem

• learned query optimizer
• dynamic execution environment
• input statistics
• model refinement
• workload variability

Innovation

• learned query optimization
• statistics-free plan encoding
• execution environment modeling
• domain adaptation
• cost estimation