CONCORD: Asynchronous Sparse Aggregation for Device-Cloud RAG under Document Isolation

2026-06-13

AI Summary

This work addresses the throughput bottleneck in device–cloud collaborative Retrieval-Augmented Generation (RAG), where privacy and policy constraints confine private documents to the device while public knowledge resides in the cloud, leading existing approaches to suffer from frequent synchronization and dense evidence transmission. The authors propose CONCORD, a novel framework that achieves asynchronous sparse aggregation for the first time: it employs a wait-debt control mechanism to dynamically decide whether to await cloud responses and introduces a certificate-guided minimal supplementation strategy that requests only the remote evidence strictly necessary for the current greedy decision, executing all other steps locally. Evaluated on Natural Questions and WikiText-2, CONCORD improves end-to-end throughput by 1.66× and 2.15×, respectively, reduces per-token communication by over two orders of magnitude, and maintains comparable answer quality and perplexity.

Abstract

Retrieval-augmented generation (RAG) has emerged as a pivotal technique for improving language models by incorporating external knowledge at inference time. As device-cloud collaborative inference makes it feasible to deploy small language models on edge devices, a new setting arises in which private documents remain on the device and public knowledge resides in the cloud. Privacy and policy constraints often forbid raw document exchange, creating a document-isolated dual-end RAG setting. However, existing methods rely on frequent remote synchronization and dense evidence transfer, limiting throughput under realistic latency and bandwidth conditions. To address this issue, we propose CONCORD, an asynchronous sparse aggregation framework for dual-end RAG under document isolation. CONCORD treats the cloud as an asynchronously arriving evidence source rather than a continuously synchronized co-generator. Specifically, we introduce waiting debt control to decide whether each decoding step should continue waiting for remote participation based on the observed return of waiting. We also design a certificate-guided minimal supplementation mechanism that requests only the remote evidence needed to determine the current greedy decision. Steps that consult the cloud preserve the same greedy token as dense dual-end aggregation, while the remaining steps commit locally without remote evidence. Experiments on Natural Questions and WikiText-2 show that CONCORD improves end-to-end throughput over baselines by $1.66\times$ and $2.15\times$, respectively, while reducing per-token communication by over two orders of magnitude and maintaining comparable answer quality and perplexity.

Problem

• retrieval-augmented generation
• document isolation
• device-cloud collaboration
• asynchronous aggregation
• sparse communication

Innovation

• asynchronous sparse aggregation
• device-cloud RAG
• document isolation
• waiting debt control
• certificate-guided supplementation