data collection and ingestion

This work addresses the challenge of reconciling high throughput and low query latency in traditional ETL pipelines when processing continuously arriving fresh data, where unpredictable preprocessing operations often create bottlenecks. The authors propose Fluid ETL Pipelines, which introduce, for the first time, an elastic and non-blocking preprocessing mechanism that decouples data ingestion from transformation. By dynamically scheduling preprocessing tasks based on resource availability and user interest—without blocking data ingestion—and leveraging preemptible computing resources such as Amazon Spot instances, the approach significantly reduces operational costs. Experimental results demonstrate that Fluid ETL Pipelines substantially improve the efficiency of exploring fresh data, offering a novel direction for accelerating real-time queries and enabling adaptive preprocessing management.

To address the challenges of protocol heterogeneity, architectural rigidity, and limited scalability in cloud-edge collaborative multimedia data ingestion, this paper proposes and implements an open-source, modular multimedia ingestion system. The system innovatively decouples the ingestion pipeline into configurable, concurrently executing microservices, supporting mainstream streaming protocols—including SRT, RIST, DASH-IF LMI, and MOQT—and is designed following cloud-native principles to ensure compatibility with Kubernetes as well as lightweight edge deployments. Compared to monolithic ingestion solutions, it significantly enhances protocol adaptability, workflow customizability, and horizontal scalability. Real-world deployment evaluations demonstrate its robust stability and high resource efficiency under high-concurrency, multi-source heterogeneous scenarios. The system provides a reusable, production-ready infrastructure foundation for modern multimedia workflows.

Existing large model training data pipelines struggle to simultaneously ensure batch semantics, fault isolation, and consistency. This work proposes an agent-free, object-storage-native training data plane featuring three core innovations: a Transactional Global Batch (TGB) abstraction that guarantees training consistency, a storage-layer-embedded garbage collection mechanism aligning producer states with distributed checkpoints, and a communication-free Decentralized Adaptive Commit (DAC) algorithm. Leveraging lakehouse ACID semantics in object storage and distributed checkpointing, the system achieves higher throughput than colocated loaders and Kafka, lower read latency, and full fault isolation across 64-GPU multimodal pretraining and supervised fine-tuning (SFT) workloads.

This work addresses the non-determinism in real-time stream processing—caused by factors such as scheduling, window triggering, out-of-order data, and network jitter—which hinders auditability and reproducibility. The authors propose a lightweight blockchain-based provenance architecture that, for the first time, integrates blockchain as a cryptographic anchor into Kafka Streams. Instead of storing raw payloads on-chain, the system records only Merkle roots of windowed data along with offset metadata, preserving data privacy and system performance while enabling verifiable integrity, sequence consistency, and analytical correctness of processing results. Experiments using a real-world weather dataset from Berlin demonstrate that the approach supports fully deterministic replay, incurs linearly scalable verification overhead, and achieves high-throughput blockchain integration, effectively meeting the demands of real-time stream processing scenarios.

This work addresses the practical challenges of deploying machine learning models in real-world settings, where heterogeneous data protocols, non-standard formats, and infrastructure constraints often necessitate redundant construction of integration pipelines. To overcome these issues, we propose SMOCS—a containerized, streaming ML system built on Apache Kafka—that decouples infrastructure from application logic through layered abstraction and employs a three-threaded agent architecture to separate data ingestion, online training, and real-time inference. The framework enables configuration-driven, no-code deployment, offering platform independence, fault isolation, and horizontal scalability, thereby significantly lowering the barrier to entry for domain experts. SMOCS has been open-sourced on the Jefferson Lab GitHub repository and demonstrates both continuous online learning capability and strong engineering practicality.

This work addresses the performance limitations of modern observability platforms under high-concurrency, compute-intensive queries, where repeated filtering over high-frequency data incurs significant overhead. The authors propose FluxSieve, the first unified data plane architecture that integrates pull-based analytics and push-based stream processing by embedding a lightweight streaming precomputation and filtering layer directly into the ingestion path. Key innovations include a scalable mechanism supporting multi-pattern concurrent matching and dynamic rule updates, along with seamless integration with Apache Pinot and DuckDB. Experimental results demonstrate that FluxSieve achieves orders-of-magnitude performance improvements across diverse systems and query workloads, while introducing only negligible storage overhead and minimal computational burden.

This work addresses the limitations of traditional data transfer tools, which typically support only streaming or batch modes in isolation and thus struggle to meet the unified management demands of heterogeneous data migration across multi-cloud environments. Building upon the Skyplane framework, we propose the first control plane that natively unifies both streaming and batch data transfers. Our system employs a URI-driven automatic routing mechanism to intelligently select the optimal transfer mode and integrates record-level structured data ingestion with chunked binary object transmission. Evaluated in an environmental monitoring scenario, the system demonstrates significant reductions in operational complexity while achieving high-throughput performance across regions, offering an efficient and unified solution for data migration in heterogeneous cloud infrastructures.

This work proposes a serverless real-time data processing framework that integrates the MapReduce programming model with Function-as-a-Service (FaaS) to address the demand for efficient handling of massive, real-time data streams in modern logistics systems. Built on Kubernetes and Knative, the framework employs an event-driven architecture composed of five loosely coupled services for data ingestion, aggregation, and analysis. It leverages Apache Kafka for event transport, Redis for metadata management, and AWS S3 for durable storage. By innovatively combining MapReduce’s batch-processing semantics with FaaS’s elastic scaling capabilities, the system supports on-demand autoscaling and scale-to-zero functionality. Experimental results demonstrate that the proposed approach achieves low-latency processing while significantly improving resource efficiency, thereby fulfilling the requirements of highly elastic and highly available data pipelines.

This work addresses the high network costs and operational overhead incurred by cross-availability-zone repartitioning in large-scale stream processing systems. It introduces, for the first time, cloud object storage into the stream shuffle pipeline through a pluggable architecture built atop Kafka Streams: upstream operators batch-write repartitioning data to object storage and emit lightweight notifications, while downstream operators fetch data on demand. By integrating configurable batching and distributed caching, the approach ensures consistency without requiring modifications to Kafka or underlying infrastructure. Experimental evaluation on AWS Kubernetes clusters demonstrates over 40× reduction in repartitioning costs, 95th-percentile latency under 2 seconds, and throughput exceeding 2 GiB/s, substantially enhancing scalability and efficiency.

This study develops and evaluates a systematic methodology for constructing news datasets from Google News, combining automated web scraping, large language model (LLM)-based metadata extraction, and SCImago Media Rankings enrichment. Using the IFMIF-DONES fusion energy project as a case study, we implemented a five-stage data collection pipeline across 81 region-language combinations, yielding 1,482 validated records after a 56% noise reduction. Results are compared against two licensed press databases: MyNews (2,280 records) and ProQuest Newsstream Collection (148 records). Overlap analysis reveals high complementarity, with 76% of Google News records exclusive to this platform. The dataset captures content types absent from proprietary databases, including specialized outlets, institutional communications, and social media posts. However, significant methodological challenges emerge: temporal instability requiring synchronic collection, a 100-result cap per query demanding multi-stage strategies, and unexpected noise including academic PDFs, false positives, and pornographic content infiltrating results through black hat SEO techniques. LLM-assisted extraction proved effective for structured articles but exhibited systematic hallucination patterns requiring validation protocols. We conclude that Google News offers valuable complementary coverage for communication research but demands substantial methodological investment, multi-source triangulation, and robust filtering mechanisms to ensure dataset integrity.