elasticsearch

This work addresses the challenge of reconciling high throughput and low query latency in traditional ETL pipelines when processing continuously arriving fresh data, where unpredictable preprocessing operations often create bottlenecks. The authors propose Fluid ETL Pipelines, which introduce, for the first time, an elastic and non-blocking preprocessing mechanism that decouples data ingestion from transformation. By dynamically scheduling preprocessing tasks based on resource availability and user interest—without blocking data ingestion—and leveraging preemptible computing resources such as Amazon Spot instances, the approach significantly reduces operational costs. Experimental results demonstrate that Fluid ETL Pipelines substantially improve the efficiency of exploring fresh data, offering a novel direction for accelerating real-time queries and enabling adaptive preprocessing management.

This work addresses the detrimental impact of microservice code smells on system maintainability and performance, exacerbated by existing detection tools that produce opaque results and lack integration with runtime observability infrastructures. To bridge this gap, the authors propose SmellDoc—a customized framework built upon the Elastic Stack that seamlessly integrates static code analysis, runtime monitoring, and a knowledge base encompassing 84 distinct smells through a dedicated Kibana plugin. SmellDoc enables cross-category detection and visualization of 24 representative microservice smells. Its key innovation lies in the introduction of two novel collectors: the Custom-Business-Collector and the Re-integration Collector, which effectively fuse business-level metrics with heterogeneous runtime telemetry data. Case studies demonstrate that SmellDoc substantially enhances runtime observability, accelerates fault localization, and ultimately safeguards service quality.

This work addresses the challenge of meeting deadlines while minimizing costs for concurrent streaming queries in cluster environments under dynamic workloads and unpredictable query arrivals. The paper proposes an intermittent query scheduling framework tailored for elastic parallel execution, which, to the best of our knowledge, is the first to integrate elastic resource provisioning into intermittent query processing. By dynamically scaling cluster nodes up or down, the approach simultaneously satisfies windowed query deadlines and reduces resource expenditure. Implemented on Apache Spark and deployed on AWS EMR, the system combines elastic computing with batch-oriented scheduling algorithms. Experimental evaluations on TPC-H and Yahoo Streaming Benchmark datasets demonstrate significant improvements over both static configurations and Spark Streaming, achieving superior timeliness and cost efficiency.

To address the challenge of processing multi-source, heterogeneous data in social networks, this paper proposes a unified batch-stream-graph analytics framework built upon the Hadoop-Spark ecosystem. The method systematically integrates Hive (for SQL-based batch processing), HBase (for low-latency key-value lookups), and GraphX (for scalable graph computation) under a single Spark execution layer. It supports three core analytical tasks: user influence assessment, high-frequency term statistics, and community relationship mining. Leveraging HDFS for distributed storage, YARN for resource orchestration, and multi-language APIs, the framework achieves loosely coupled integration of computation and storage. End-to-end experiments on real-world social datasets demonstrate that the hybrid architecture accelerates complex relational analysis by 1.8–3.2× compared to single-component baselines, significantly improving both processing efficiency and system flexibility.

Efficiently mapping structured array aggregations—such as circular, grid, hierarchical, and sliding-window aggregations—in array databases (e.g., SciDB) onto MapReduce remains challenging due to semantic mismatches between array-centric operations and key-value abstractions. Method: This paper introduces the first array-semantic-aware AQL-to-MapReduce automatic translation framework. It models array-specific structural aggregations as rewriteable semantic rules, integrates aggregation-aware data partitioning, and generates multi-stage MapReduce jobs—fully supporting user-defined aggregation functions without modifying the underlying system. Contribution/Results: Experimental evaluation shows that the generated code achieves up to 10.84× speedup over manual MapReduce implementations, while guaranteeing semantic correctness and drastically reducing development complexity. The framework bridges a critical gap between high-level array query languages and distributed compilation optimizations, enabling scalable, declarative array analytics on MapReduce infrastructures.

This work proposes a serverless real-time data processing framework that integrates the MapReduce programming model with Function-as-a-Service (FaaS) to address the demand for efficient handling of massive, real-time data streams in modern logistics systems. Built on Kubernetes and Knative, the framework employs an event-driven architecture composed of five loosely coupled services for data ingestion, aggregation, and analysis. It leverages Apache Kafka for event transport, Redis for metadata management, and AWS S3 for durable storage. By innovatively combining MapReduce’s batch-processing semantics with FaaS’s elastic scaling capabilities, the system supports on-demand autoscaling and scale-to-zero functionality. Experimental results demonstrate that the proposed approach achieves low-latency processing while significantly improving resource efficiency, thereby fulfilling the requirements of highly elastic and highly available data pipelines.

This work addresses the operational complexity and architectural coupling introduced by conventional vector similarity search systems in compute-storage disaggregated environments, which typically rely on a separate indexing layer. To resolve this, the authors propose deeply integrating a distributed approximate nearest neighbor (ANN) index into the Apache Iceberg table format. The approach leverages Puffin sidecar files to store sharded Vamana graphs and employs Iceberg’s snapshot mechanism to atomically bind indexes with data, enabling versioning and time travel. A coordinator-executor protocol is introduced, featuring a hierarchical probing strategy where coordinators cache compact centroid indexes while executors store large graphs on SSDs. The design reuses Iceberg’s REST catalog optimistic concurrency control for index commits. This solution is the first to host billion-scale vector graph indexes within Puffin, achieving a favorable trade-off between recall and latency while preserving Iceberg’s native transactional capabilities and significantly reducing system complexity.

This work addresses severe resource contention between indexing and querying in large-scale search engines, which leads to increased query latency. To mitigate this, the authors propose ScaleSearch, a novel architecture that incorporates five core mechanisms: node-level read-write separation, decoupling of compute and storage, fully in-memory indexing, log-structured write paths, and field-level update routing. These mechanisms collectively enable O(1) in-place updates for scalar fields and isolated processing for full-text fields. ScaleSearch substantially alleviates read-write interference, achieving both low query latency and high index freshness. The architecture thus provides an efficient and real-time system foundation for emerging hybrid retrieval scenarios combining vector and full-text search.

This work addresses the challenge of efficiently constructing and querying inverted indexes over large-scale text corpora by designing and implementing a high-performance, memory-safe, and scalable inverted index library in Rust. Leveraging Rust’s zero-cost abstractions, concurrency safety guarantees, and expressive trait-based generics, the system flexibly integrates multiple classical inverted indexing techniques and supports efficient full-text retrieval algorithms. Experimental evaluation across several standard datasets and query workloads demonstrates that the proposed library achieves up to twice the query performance of state-of-the-art alternatives, substantially enhancing both retrieval efficiency and practical applicability.