SmartRAG: Native Graph-Based RAG for Mobile Device

2026-07-16

AI Summary

Deploying large language models (LLMs) on mobile devices demands a balance among privacy preservation, low latency, and offline availability, yet is constrained by limited edge hardware resources. This work proposes the first fully on-device intelligent assistant framework, integrating four core modules—perception, memory, focus, and reasoning—enabled by EvoNER, a continual-learning named entity recognizer, and MRGraph, a three-layer provenance-aware knowledge graph that supports entity expansion and multi-hop reasoning without model retraining. The system employs a hybrid retrieval mechanism combining graph traversal, lexical matching, and dense semantic search to invoke a quantized LLM only for high-value tasks. Evaluated on four multi-hop question answering benchmarks, the approach achieves performance comparable to models 18 times larger using only a 1.7B-parameter quantized LLM, satisfying typical smartphone memory and latency constraints.

Abstract

Deploying large language models (LLMs) as personal assistants on mobile devices demands privacy, low latency, and offline availability, yet the computational cost of giant models clashes with strict edge-hardware budgets. We argue that this tension cannot be resolved by model compression alone; it requires decomposing on-device intelligence into complementary functional roles. We present SmartRAG, a fully on-device framework that organizes an intelligent assistant around four coordinated modules -- Perception, Memory, Focus, and Thinking. At the core of SmartRAG is EvoNER, a continually learnable named-entity recognizer that incrementally expands its label inventory through teacher-distilled updates, enabling the system to absorb previously unseen entity types without retraining the backbone LLM. Extracted knowledge is stored in MRGraph, a three-layer provenance-preserving knowledge graph, and retrieved at query time through a hybrid pipeline combining graph traversal, lexical matching, and dense semantic search. The on-device LLM is invoked only for high-value semantic operations -- labeling, planning, and answer synthesis -- keeping inference costs bounded. Experiments on four QA benchmarks (TriviaQA, Natural Questions, HotpotQA, MultiHopQA) show that SmartRAG with a quantized 1.7B-parameter backbone achieves multi-hop reasoning performance competitive with models up to 18$\times$ larger, while running entirely on commodity smartphones within practical memory and latency envelopes.

Problem

• on-device AI
• large language models
• mobile deployment
• resource constraints
• privacy

Innovation

• on-device RAG
• continual learning
• knowledge graph
• named entity recognition
• mobile LLM