Icicle: Scalable Metadata Indexing and Real-Time Monitoring for HPC File Systems

2026-04-11

AI Summary

This work addresses the growing challenge of metadata management in modern high-performance computing (HPC) environments, where billions of files and hundreds of petabytes of data overwhelm traditional tools that struggle to support real-time, heterogeneous, and dynamically evolving query and monitoring demands. The paper proposes the first scalable metadata indexing and monitoring framework integrating a stream processing architecture, combining snapshot-based bulk ingestion with event-driven real-time synchronization to achieve high-throughput, low-latency unified indexing. Built on Apache Kafka and Flink, the system leverages native event collection mechanisms from Lustre and IBM Storage Scale and maintains dual indices to enable multi-granularity queries ranging from single-file lookups to aggregate statistics. Experiments on production-scale HPC datasets demonstrate an order-of-magnitude improvement in throughput over existing approaches, while offering flexible trade-offs among consistency, latency, and metadata freshness.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Time-Series/Data Streams
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Security and Privacy: Data transparency and provenance

Abstract

Modern HPC file systems can contain billions of files and hundreds of petabytes of data, making even simple questions increasingly intractable to answer. Traditional file system utilities such as find and du fail to scale to these sizes. While external indexing tools like GUFI and Brindexer improve query performance, they remain batch-oriented and unsuitable for heterogeneous, rapidly evolving environments. We present Icicle, a scalable framework for continuous file system metadata indexing and monitoring. Icicle maintains a unified, up-to-date, and queryable view of file system state while supporting both periodic snapshot-based ingestion for bulk metadata updates and event-based ingestion for real-time synchronization from production systems such as Lustre and IBM Storage Scale. Built on Apache Kafka and Apache Flink, Icicle provides high-throughput, fault-tolerant, and horizontally scalable ingestion of metadata events into two complementary search indexes, enabling both individual file discovery and aggregate summary statistics by user, group, and directory. This architecture enables efficient support for both coarse-grained administrative queries and interactive analytics over billions of objects. Our experimental evaluation on production-scale HPC datasets demonstrates order-of-magnitude throughput improvements over existing monitoring and indexing approaches, with tunable options for balancing consistency, latency, and metadata freshness.

Problem

• HPC file systems
• metadata indexing
• real-time monitoring
• scalability
• file system query

Innovation

• real-time metadata indexing
• scalable file system monitoring
• event-driven ingestion
• HPC storage systems
• stream processing architecture