CCD-Level and Load-Aware Thread Orchestration for In-Memory Vector ANNS on Multi-Core CPUs

2026-05-11

AI Summary

This work addresses the inefficiency of vector approximate nearest neighbor search (ANNS) on multi-core CPUs, which struggles to scale effectively due to memory latency, compute-intensive distance calculations, and insufficient exploitation of Cache-Coherent Domain (CCD) architecture characteristics. To overcome these limitations, the authors propose a CCD-aware, load- and hardware-conscious thread scheduling framework that uniformly supports both inter-query and intra-query parallelism for HNSW and IVF algorithms. By integrating cache-friendly task assignment, load-adaptive mapping, and a CCD-aware work-stealing mechanism, the framework substantially improves resource utilization. Evaluated on real-world workloads from Xiaohongshu, the system achieves up to 3.7× higher throughput, reduces P50/P999 latency by 30–90%, decreases cache miss rates by 6–30%, and cuts CPU stall cycles by 20–80%.

Abstract

Vector approximate nearest neighbor search (ANNS) underpins search engines, recommendation systems, and advertising services. Recent advances in ANNS indexes make CPU a cost-effective choice for serving million-scale, in-memory vector search, yet per-core throughput remains constrained by memory access latency of vector reading and the compute intensity of distance evaluations in production deployments. With the growing scale of the business and advances in hardware, modern CCD-based multi-core CPUs have been widely deployed for high throughput in our services. However, we find that simply increasing core counts does not yield optimal performance scaling. To improve the efficiency of more cores from the CCD-based architecture, we analyze the distributions of real-world requests in our production environments. We observe high access locality in vector search in our online services and low cache utilization, resulting from overlooking the multi-chiplet nature of CCD based CPUs. Hence, we propose a workload- and hardware-aware thread orchestration framework at CCD-level that (i) provides a uniform interface for both inter-query parallel HNSW search and intra-query parallel IVF search, (ii) achieves cache-friendly and workload-adaptive mapping of task dispatching, and (iii) employs CCD-aware task stealing to address load imbalance. Applied to real production workloads from search, recommendation, and advertising services of Xiaohongshu (RedNote), our approach delivers up to 3.7x higher throughput and 30-90% reductions in P50 and P999 latency. In detail, compared with the original framework, the cache-miss ratio decreases by 6-30%, and the total CPU stall is reduced by 20-80%.

Problem

• vector ANNS
• multi-core CPUs
• CCD architecture
• cache locality
• load imbalance

Innovation

• CCD-aware scheduling
• in-memory vector ANNS
• cache-friendly task mapping
• load-aware thread orchestration
• multi-core CPU optimization