OTRO: Oblivious Tokenization Path with Square-Root ORAM

2026-06-15

AI Summary

This work addresses the privacy leakage in large language model (LLM) tokenizers caused by memory access patterns during CPU-side table lookups, which can expose user prompts. While existing Oblivious RAM (ORAM) solutions mitigate this risk, they incur prohibitive latency overheads. To overcome this limitation, the authors propose a low-latency, high-efficiency oblivious tokenization pipeline that constructs a replicated instance pool based on Square-Root ORAM. The design incorporates epoch-based rotation and boundary padding to reduce reconstruction frequency and introduces a KV cache-aware chunking mechanism that overlaps tokenization with GPU prefilling. Evaluated in a TDX+CVM+H100 environment, the approach incurs ≤4.5% first-token latency overhead, keeps tokenization latency below 10% of total inference time, and adds less than 0.5 GB of memory overhead, effectively suppressing access pattern leakage across multiple models.

Abstract

The CPU-side large language model (LLM) tokenizer is a critical security gap in LLM serving through a confidential computing stack with CPU and GPU trusted execution environments (TEEs). Tokenizers converts the prompts through table-driven lookups, and the resulting memory access patterns are a powerful source of side-channel leakage. Recent work demonstrates end-to-end recovery of user prompts from tokenizer access pattern on production Intel TDX. However, a drop-in use of the popular tree-based Oblivious RAMs (e.g., PathORAM) to prevent access-pattern leakage introduces $\sim$13$\times$ tokenizer slowdown, resulting in 10-58% higher time-to-first-token (TTFT). In this paper, we present OTRO, an efficient, oblivious tokenization path tailored to latency-critical LLM serving. OTRO relies on square-root ORAM for fast single-access lookups, but avoids its prohibitive $O(N\log^2N$) rebuild cost every $\sqrt{N}$ accesses through three key innovations. First, OTRO provides a pool of replicated square-root ORAM instances that utilize the read-only nature of tokenizer table. Second, an epoch-based rotation policy decouples accesses from rebuilds and pads each epoch with dummy accesses to its boundaries, minimizing observable information. Lastly, chunked KV-cache-aware tokenization further overlaps rebuilds with GPU prefill and minimizes the instance count. Implemented as modules in HuggingFace Tokenizers and nano-vLLM, running within a TDX-enabled CVM with an NVIDIA H100 GPU, OTRO limits TTFT overhead to at most 4.5%, keeps tokenizer-induced latency under 10\% of total TTFT, and adds less than 0.5 GB of memory overhead while reducing the tokenizer's observable leakage across various model families and sizes.

Problem

• tokenizer
• side-channel leakage
• trusted execution environment
• Oblivious RAM
• LLM serving

Innovation

• Oblivious RAM
• Tokenizer Security
• Confidential Computing
• Square-Root ORAM
• Side-Channel Mitigation