Rethinking Burst Buffer Optimization: Enabling Layout Heterogeneity via Hybrid Analysis and LLM Guidance

2026-06-19

AI Summary

Existing burst buffer file systems suffer from performance degradation due to rigid, fixed data layouts that fail to adapt to diverse application I/O behaviors. This work proposes Proteus, a system that treats data layout as a first-class optimization target by integrating static code analysis with lightweight runtime probes to reconstruct an application’s I/O semantic intent during a single pre-execution pass. Guided by a large language model, Proteus makes layout decisions without requiring training or intrusive profiling. Evaluated on representative HPC workloads, Proteus achieves a layout decision accuracy of 91.30% and delivers speedups of up to 3.24× in write-intensive scenarios and 2.9× in metadata-intensive workloads.

Abstract

Burst buffers (BBs) are essential for mitigating I/O bottlenecks in modern HPC systems. However, existing BB file systems often suffer from structural performance degradation due to fixed data layouts that fail to align with diverse application behaviors. While current machine-learning-based optimizations focus primarily on tuning storage stack parameters for a given layout, they offer diminishing returns when a fundamental mismatch exists between I/O patterns and the underlying data organization. Furthermore, these approaches typically incur prohibitive costs due to extensive training or intrusive profiling. To bridge this gap, we present Proteus, a semantic-aware BB system that treats data layout as a first-class optimization dimension. The core insight of Proteus is that application I/O intent can be reconstructed by synergetically combining static code structures with lightweight runtime signals. Through a hybrid pipeline and a single execution probe, Proteus extracts latent semantic cues to determine the optimal layout prior to production runs-eliminating the need for prior training or exhaustive profiling. Evaluation with representative HPC workloads shows that Proteus achieves 91.30\% decision accuracy, delivering up to 3.24$\times$ and 2.9$\times$ speedups for write-intensive and metadata-intensive workloads, respectively.

Problem

• Burst Buffer
• Data Layout
• I/O Optimization
• HPC Systems
• Performance Degradation

Innovation

• burst buffer
• data layout optimization
• hybrid analysis
• LLM guidance
• semantic-aware I/O