Atompack: A Storage and Distribution Layer for Read-Heavy Atomistic ML Training Datasets

2026-06-29

AI Summary

This work addresses the inefficiency of conventional storage formats in supporting high-throughput, randomly shuffled reads and compact distribution of complete molecular records for atomistic machine learning. To overcome these limitations, the authors propose Atompack, a novel append-optimized storage and distribution format that treats entire molecules as atomic units. Atompack employs an immutable index and memory-mapped I/O, eliminating field-wise chunking and runtime record reconstruction. Experimental results on a representative task involving 64-atom systems demonstrate that Atompack achieves a 96× speedup in shuffled read performance and a 79% reduction in data volume compared to ASE LMDB, substantially improving both training data construction efficiency and access performance.

Abstract

Atomistic machine learning datasets are increasingly used for training: large immutable snapshots are read repeatedly, shuffled across epochs, staged across clusters' storage systems, and republished as reusable scientific artifacts. This workload differs from interactive scientific curation, where mutable records and ad hoc inspection are often more important than random indexed throughput. We present Atompack, an append-oriented storage format and distribution layer designed around a simple workload: training pipelines usually consume complete molecular records, while the order of records is randomized by the learning algorithm. Atompack appends records efficiently during dataset construction, then commits an immutable index and serves records through a memory-mapped read path optimized for training. We compare Atompack with HDF5, LMDB, and ASE baselines representing array stores, key-value records, serialized records, and object-oriented databases. The benchmarks measure sequential reads, shuffled reads, shared-filesystem behavior, write throughput, and artifact size. On a representative 64-atom workload, Atompack is 96x faster than ASE LMDB on shuffled training-style reads while producing artifacts about 79\% smaller. The results indicate that serving complete molecule records, rather than field chunks or reconstructed objects, improves shuffled training throughput while keeping artifacts compact enough for public distribution.

Problem

• atomistic machine learning
• training datasets
• storage format
• data distribution
• shuffled reads

Innovation

• Atompack
• atomistic machine learning
• append-oriented storage
• memory-mapped I/O
• immutable dataset