Bringing Data Transformations Near-Memory for Low-Latency Analytics in HTAP Environments

2026-01-18

AI Summary

This work addresses the high analytical latency and severe resource contention in HTAP systems caused by traditional ETL pipelines, which entail frequent data movement. To overcome these limitations, the authors propose offloading data transformation logic to an intelligent storage layer that leverages near-data or in-storage computing capabilities to perform format conversion and preprocessing directly at the storage tier. This approach eliminates the overhead of data migration and significantly reduces interference with foreground transactional workloads, thereby enhancing both the performance and real-time responsiveness of analytical queries. Experimental results demonstrate that the proposed architecture achieves a highly reusable, low-latency data processing paradigm under mixed workloads across multiple execution engines, offering an efficient and scalable storage-compute co-design for HTAP systems.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Hardware-aware ML
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

In this paper we propose an approach for executing data transformations near- or in-storage on intelligent storage systems. The currently prevailing approach of extracting the data and then transforming it to a target format suffers degraded performance during transformation and causes heavy data movement. Our results show robust performance of foreground workloads and lower resource contention. Our vision draws architectural opportunities in multi-engine and multi-system settings, as well as for reuse.

Problem

• data transformation
• near-memory computing
• HTAP
• data movement
• storage systems

Innovation

• near-memory computing
• data transformation
• intelligent storage
• HTAP
• data movement reduction