OpenMLDB: A Real-Time Relational Data Feature Computation System for Online ML

2025-01-15

AI Summary

In online machine learning, inconsistencies between offline training and online serving feature computation—coupled with inefficient temporal multi-table feature processing—severely hinder real-time performance. To address this, we propose a real-time relational feature computation system tailored for online learning. Our approach introduces: (1) a unified query plan generator to ensure end-to-end feature consistency across the ML lifecycle; (2) a pre-aggregation mechanism with data-adaptive windowing to overcome performance bottlenecks in long-window and multi-table join scenarios; and (3) a time-aware indexing scheme and compact in-memory data format. Built atop a unified SQL engine, the system supports window-level parallelism, time-aware skew mitigation, and streaming memory indexing. Empirical evaluation demonstrates millisecond-scale latency—10× to 100× faster than Flink and Spark—70% lower deployment overhead, and production deployment across 100+ live services. The open-source implementation has garnered 1.6k GitHub stars and 150+ contributors.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Efficient and consistent feature computation is crucial for a wide range of online ML applications. Typically, feature computation is divided into two distinct phases, i.e., offline stage for model training and online stage for model serving. These phases often rely on execution engines with different interface languages and function implementations, causing significant inconsistencies. Moreover, many online ML features involve complex time-series computations (e.g., functions over varied-length table windows) that differ from standard streaming and analytical queries. Existing data processing systems (e.g., Spark, Flink, DuckDB) often incur multi-second latencies for these computations, making them unsuitable for real-time online ML applications that demand timely feature updates. This paper presents OpenMLDB, a feature computation system deployed in 4Paradigm's SageOne platform and over 100 real scenarios. Technically, OpenMLDB first employs a unified query plan generator for consistent computation results across the offline and online stages, significantly reducing feature deployment overhead. Second, OpenMLDB provides an online execution engine that resolves performance bottlenecks caused by long window computations (via pre-aggregation) and multi-table window unions (via data self-adjusting). It also provides a high-performance offline execution engine with window parallel optimization and time-aware data skew resolving. Third, OpenMLDB features a compact data format and stream-focused indexing to maximize memory usage and accelerate data access. Evaluations in testing and real workloads reveal significant performance improvements and resource savings compared to the baseline systems. The open community of OpenMLDB now has over 150 contributors and gained 1.6k stars on GitHub.

Problem

• Online Machine Learning
• Feature Inconsistency
• Computational Efficiency

Innovation

• Online Learning
• Unified Computing Schema
• Efficient Data Processing