FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation

2026-06-22

AI Summary

Deploying large language model (LLM) inference on ARM TrustZone faces significant performance overhead due to rigid security resource isolation and inefficient management. To address this, this work proposes FlexServe, a novel system that decouples access control from management authority for secure resources. FlexServe introduces reclaimable secure memory (Flex-Mem) and secure NPU (Flex-NPU), and establishes a cooperative memory management framework between the secure world and the normal world. This design preserves TrustZone’s strong isolation guarantees while substantially enhancing resource management flexibility and inference efficiency. Experimental results demonstrate that FlexServe reduces average time-to-first-token latency by 10.05× compared to baseline TrustZone implementations and achieves a 2.44× improvement over existing optimized approaches.

Abstract

Device-side Large Language Models (LLMs) have grown explosively, offering stronger privacy and higher availability than their cloud-side counterparts. During LLM inference, both the model weights and the user data are valuable, and attackers may compromise the OS kernel to steal them. ARM TrustZone is the de facto hardware-based isolation technology on mobile devices, used to protect sensitive applications from a compromised OS. However, protecting LLM inference with TrustZone incurs significant overhead to both the secure inference and the normal aplications, due to two challenges: the inflexible resource isolation and the inefficient secure resource management. To address these challenges, this paper presents FlexServe, a fast and secure LLM inference system for mobile devices. The key idea is to decouple the access permission from the management permission of secure resources, so that the normal-world OS cannot access them but can still manage them as usual. First, FlexServe introduces a Recallable Resource Isolation mechanism to construct Recallable Secure Memory (Flex-Mem) and a Recallable Secure NPU (Flex-NPU). They can only be accessed by the secure world, but can be efficiently allocated and reclaimed by the normal-world OS. Based on them, FlexServe further introduces a FlexServe Framework to run secure LLM inference in the secure world. It works together with the normal-world OS to perform cooperative secure memory management. We implement a prototype of FlexServe and compare it with two TrustZone-based strawman designs. The results show that FlexServe achieves average TTFT speedups of 10.05X over the strawman and 2.44X over an optimized strawman.

Problem

• LLM inference
• ARM TrustZone
• resource isolation
• secure memory management
• mobile devices

Innovation

• Flexible Resource Isolation
• ARM TrustZone
• Secure LLM Inference
• Recallable Secure Memory
• Cooperative Memory Management