Stochastic SketchRefine: Scaling In-Database Decision-Making under Uncertainty to Millions of Tuples

2024-11-26

arXiv.org

AI Summary

This work addresses the optimization of large-scale Stochastic Package Queries (SPQ) over uncertain data—specifically, efficiently computing integer solutions that maximize expected utility while satisfying risk constraints across hundreds of millions of tuples under high-variance stochastic attributes. We propose: (1) Risk Constraint Linearization (RCL), a technique that transforms scenario-dependent stochastic optimization into a scenario-independent integer linear program by linearizing probabilistic risk constraints; and (2) Stochastic SketchRefine, a divide-and-conquer framework integrating randomized sketching with iterative refinement, enabling sub-second SPQ evaluation at million-scale for the first time. Compared to state-of-the-art methods, our approach achieves over two orders-of-magnitude speedup, maintains near-optimal solution quality, and provides rigorous risk guarantees. This significantly advances real-time, risk-aware decision-making capabilities within database systems.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Distributed Search

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Large-scale security measurements

Abstract

Decision making under uncertainty often requires choosing packages, or bags of tuples, that collectively optimize expected outcomes while limiting risks. Processing Stochastic Package Queries (SPQs) involves solving very large optimization problems on uncertain data. Monte Carlo methods create numerous scenarios, or sample realizations of the stochastic attributes of all the tuples, and generate packages with optimal objective values across these scenarios. The number of scenarios needed for accurate approximation - and hence the size of the optimization problem when using prior methods - increases with variance in the data, and the search space of the optimization problem increases exponentially with the number of tuples in the relation. Existing solvers take hours to process SPQs on large relations containing stochastic attributes with high variance. Besides enriching the SPaQL language to capture a broader class of risk specifications, we make two fundamental contributions towards scalable SPQ processing. First, to handle high variance, we propose risk-constraint linearization (RCL), which converts SPQs into Integer Linear Programs (ILPs) whose size is independent of the number of scenarios used. Solving these ILPs gives us feasible and near-optimal packages. Second, we propose Stochastic SketchRefine, a divide and conquer framework that breaks down a large stochastic optimization problem into subproblems involving smaller subsets of tuples. Our experiments show that, together, RCL and Stochastic SketchRefine produce high-quality packages in orders of magnitude lower runtime than the state of the art.

Problem

• Scaling stochastic package queries to millions of tuples efficiently
• Handling high variance in uncertain data optimization problems
• Reducing runtime for large stochastic optimization problems significantly

Innovation

• Risk-constraint linearization converts SPQs to ILPs
• Stochastic SketchRefine divides large problems into subproblems
• Combined approach reduces runtime significantly