Decoupled I/O-Dominant Pipelines for Large-Scale Whole-Slide Image Embedding Extraction

2026-08-27

AI Summary

该论文针对大规模全切片图像处理中的I/O和协调开销问题，提出了一种解耦的I/O感知流水线方法，通过分阶段处理提高效率。

Abstract

Whole-slide images (WSIs) are central to computational pathology but are prohibitively large, making patch-based processing the practical unit for foundation model inference. At scale, however, generating and handling massive numbers of patches on quickly introduces significant I/O and orchestration overhead, often dominating end-to-end performance. We present a decoupled, I/O-aware pipeline for large-scale WSI embedding extraction that decomposes the workflow into three stages: (1) patch generation and staging, (2) embarrassingly parallel embedding inference, and (3) sharded vector database ingestion. This design isolates data movement from compute, enabling efficient patch delivery, scalable multi-node inference with minimal communication. The resulting system produces a distributed vector database where embeddings are persistently coupled with rich metadata (e.g., patient, slide, and patch attributes), enabling efficient filtering, retrieval, and downstream reuse. This representation database is compact and reusable for tasks such as retrieval, classification, and few-shot learning, particularly benefiting low-resource environments. We show that decoupling I/O, computation, and ingestion enables high-throughput WSI embedding extraction at scale. By characterizing the scaling envelope, we demonstrate that storage dominates beyond moderate concurrency, reframing WSI embedding extraction as a data-centric systems problem rather than a purely compute-bound workload.

Problem

• Whole-slide images
• I/O overhead
• large-scale embedding extraction
• patch-based processing
• data movement

Innovation

• Decoupled I/O-aware pipeline
• Large-scale WSI embedding extraction
• Distributed vector database
• Scalable multi-node inference
• Efficient patch delivery