Adaptive Contextual Caching for Mobile Edge Large Language Model Service

2025-01-16

AI Summary

To address the challenges of limited computational resources and bandwidth for large language model (LLM) services in mobile edge computing, as well as high retrieval latency and substantial update overhead induced by conventional RAG-based cache management, this paper proposes the first deep reinforcement learning (DRL)-based adaptive context caching framework. The framework jointly models user intent, semantic similarity, and cache-miss cost to enable dynamic, optimal caching decisions. It integrates semantic-aware caching, seamless RAG incorporation, and context-aware cache replacement. Experimental results demonstrate that the framework achieves over 80% cache hit rate within only 11 training episodes, reduces retrieval latency by 40%, and cuts local cache update overhead by 55%. These improvements significantly enhance the real-time performance and scalability of edge-deployed LLM services.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Mobile edge Large Language Model (LLM) deployments face inherent constraints, such as limited computational resources and network bandwidth. Although Retrieval-Augmented Generation (RAG) mitigates some challenges by integrating external knowledge bases, inefficient cache management can still result in high retrieval latency and frequent cache updates. To address these issues, we propose an Adaptive Contextual Caching (ACC) framework that anticipates user needs by proactively caching semantically relevant data for mobile-edge LLMs. ACC utilizes a deep reinforcement learning (DRL) module to refine cache replacement policies, balancing user context, document similarity, and the overhead associated with cache misses. Experimental results demonstrate that ACC increases cache hit rates to over 80% after only 11 training episodes, outperforming FIFO, LRU, and semantic-only caching while reducing retrieval latency by up to 40%. In particular, ACC also reduces local caching overhead (i.e., the cost of updating the cache when a miss occurs) by as much as 55%, enabling scalable, low-latency LLM services in resource-constrained edge environments.

Problem

• Large Language Models
• Cache Management
• Mobile Devices

Innovation

• Adaptive Context Caching (ACC)
• Deep Reinforcement Learning (DRL)
• Large Language Model (LLM) Optimization