🤖 AI Summary
This work addresses the operational complexity and architectural coupling introduced by conventional vector similarity search systems in compute-storage disaggregated environments, which typically rely on a separate indexing layer. To resolve this, the authors propose deeply integrating a distributed approximate nearest neighbor (ANN) index into the Apache Iceberg table format. The approach leverages Puffin sidecar files to store sharded Vamana graphs and employs Iceberg’s snapshot mechanism to atomically bind indexes with data, enabling versioning and time travel. A coordinator-executor protocol is introduced, featuring a hierarchical probing strategy where coordinators cache compact centroid indexes while executors store large graphs on SSDs. The design reuses Iceberg’s REST catalog optimistic concurrency control for index commits. This solution is the first to host billion-scale vector graph indexes within Puffin, achieving a favorable trade-off between recall and latency while preserving Iceberg’s native transactional capabilities and significantly reducing system complexity.
📝 Abstract
We describe a design pattern and concrete implementation for embedding distributed approximate nearest neighbor indexes inside the Apache Iceberg table format, using the Puffin sidecar file as the storage container and the snapshot summary as the binding mechanism. Modern analytical query engines increasingly adopt a compute disaggregated architecture: executors are stateless, scale elastically, and read all data from object storage. Adding vector similarity search to such an engine traditionally requires a dedicated index storage layer with its own lifecycle, consistency model, and operational surface breaking the disaggregation in variant. We show that the Puffin format, originally introduced portable level statistics and deletion vectors, is sufficient to carry full Vamana graphs at billion vector scale, and that linking these blobs through the existing statistics file snapshot summary property reduces ANN index management to standard Iceberg snapshot operations. We present a binary layout for sharded graph indexes inside Puffin, a coordinator executor protocol for distributed index build, probe, and incremental refresh, the integration into the existing optimistic-concurrency commit path of an Iceberg REST catalog, and a tiered probe strategy that places small centroid indexes on the coordinator and large DiskANN graphs on executor SSDs. The pattern inherits atomicity, time travel, multi engine read ability, and orphan file garbage collection from the table format at zero implementation cost. We discuss the recall/latency trade-offs introduced by the independent-shard design and quantify projected query and build performance for tables up to 109 vectors. Our implementation extends FlockDB, a distributed MPP engine built on DuckDB.