Accelerating Data Chunking in Deduplication Systems using Vector Instructions

2025-08-07

AI Summary

Content-defined chunking (CDC) is a critical step in deduplication, yet traditional algorithms require full-file scanning, creating a major performance bottleneck. This paper proposes VectorCDC—the first efficient hash-free vectorized CDC method—pioneering deep integration of SIMD instructions (e.g., SSE/AVX) into the core CDC logic without compromising deduplication ratio. Designed for cross-architecture compatibility, VectorCDC has been validated on Intel, AMD, ARM, and IBM Power processors, achieving 8.35×–26.2× throughput improvement over state-of-the-art vectorized approaches. Its key innovation lies in a lightweight, branchless vectorized sliding-window matching and chunking decision mechanism, overcoming the fundamental challenge that has historically impeded CDC vectorization.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Data Compression
• Computer Vision: Learning & Optimization for CV

Application Category

• Web Mining and Content Analysis: Content-based information diffusion
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Large-scale security measurements

Abstract

Content-defined Chunking (CDC) algorithms dictate the overall space savings that deduplication systems achieve. However, due to their need to scan each file in its entirety, they are slow and often the main performance bottleneck within data deduplication. We present VectorCDC, a method to accelerate hashless CDC algorithms using vector CPU instructions, such as SSE / AVX. Our evaluation shows that VectorCDC is effective on Intel, AMD, ARM, and IBM CPUs, achieving 8.35x - 26.2x higher throughput than existing vector-accelerated techniques without affecting the deduplication space savings.

Problem

• Accelerating slow Content-defined Chunking (CDC) algorithms
• Overcoming performance bottleneck in deduplication systems
• Enhancing throughput without compromising space savings

Innovation

• Uses vector CPU instructions for acceleration
• Applies SSE/AVX to hashless CDC algorithms
• Achieves high throughput across multiple CPU architectures