As We May Search

2026-06-28

AI Summary

Current retrieval-augmented generation systems require uploading sensitive documents to remote servers, posing significant privacy risks. This work proposes a “local-first information retrieval” paradigm that fully deploys indexing, models, and inference on the user’s device, with optional remote service invocation. The authors formally define this design paradigm for the first time and establish a system framework centered on three dimensions: privacy control, capability, and accessibility. They identify search scope—not quality—as the primary trade-off in local-first systems. Experimental results demonstrate that, on consumer-grade hardware, a local system combining dense retrieval, BM25, and HNSW indexing achieves over 91% nDCG@10 on collections of up to 100K documents, with only a 2% drop when scaled to 1M documents. Furthermore, question-answering quality using a 7B-parameter language model lags behind cloud-based baselines by merely 4 points.

Abstract

The sensitive information in personal documents, legal files, and medical records is among the most valuable things to search, yet current retrieval-augmented generation systems still require sending content to remote servers. We propose local-first IR, a design philosophy where indexes, models, and inference reside on user devices, treating remote services as optional. This paper makes four contributions: (1) a framework organizing retrieval architectures along three dimensions: privacy and control, capability, and accessibility, (2) experiments on consumer hardware across five benchmarks, scaling from 1K to 1M documents with dense retrieval, BM25, and hybrid fusion. Dense retrieval keeps over 91% nDCG@10 up to 100K documents, with approximate HNSW indexes extending this to 1M with only 2% quality loss; a 7B local language model reaches within 4 points of a cloud baseline on answer quality, (3) competing perspectives for and against local-first IR, informed by experimental evidence, and (4) a research agenda identifying open problems. The real tradeoff is scope rather than quality: what matters is what you can search, not how well you can search it.

Problem

• privacy
• retrieval-augmented generation
• local-first IR
• sensitive information
• information retrieval

Innovation

• local-first IR
• dense retrieval
• HNSW indexing
• retrieval-augmented generation
• privacy-preserving search