Problem
Research questions and friction points this paper is trying to address.
multi-modal data
storage redundancy
small-file overheads
cache-unfriendly layouts
heavy index structures
FFSlim: An Efficient and Lightweight Format for Multi-modal Data Storage and Retrieval
🤖 AI Summary
FFSlim通过统一文件格式、自适应检索机制和冗余检测模块解决了多模态数据存储冗余和I/O瓶颈问题，提高存储效率和加载吞吐量。
📝 Abstract
With the rapid expansion of large-scale media-text corpora, multi-modal datasets increasingly require efficient storage and retrieval. Existing formats such as Files, TDP, and FFRecord work adequately for uni-modal data but expose fundamental limitations in multi-modal settings, including storage redundancy, massive small-file overheads, cache-unfriendly layouts, and heavy index structures. These issues jointly inflate storage and memory usage and make I/O the dominant bottleneck in real training workloads. We present FFSlim, a lightweight format for storing and retrieving multi-modal data. FFSlim improves storage efficiency and loading throughput through three components: a unified file format that removes media duplication and avoids small-file proliferation; an adaptive retrieval mechanism that enables low-overhead pair-level access and accelerates repeated media loading; and a redundancy detection and aggregation module that converts existing datasets into the FFSlim layout. The experimental results demonstrate that FFSlim achieves 2.07x and 8.26x higher data loading and write throughput on average than the strongest baseline, with minimal storage and index overhead. Consequently, these underlying I/O accelerations enable FFSlim to reduce end-to-end training time by 5.36%-14.18% across seven diverse multi-modal models.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Unified File Format
Adaptive Retrieval Mechanism
Redundancy Detection and Aggregation
🔎 Similar Papers
No similar papers found.
