Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs

2026-04-20

AI Summary

Existing privacy evaluation methods based on indistinguishability, such as differential privacy, struggle to effectively characterize data extraction risks in large language model (LLM) APIs. This work proposes a novel privacy game framework that formally establishes, for the first time, the incomparability between indistinguishability and unextractability, and introduces $(l, b)$-unextractability to quantify the query complexity required by a black-box adversary to extract protected substrings. By integrating worst-case extraction games, rank analysis, and multi-round prefix-adaptive attack simulations—combined with greedy and probabilistic decoding strategies—the study develops upper-bound estimation methods applicable to exact, non-targeted, and approximate extraction scenarios. Empirical evaluations demonstrate that the proposed metric outperforms existing approaches across multiple LLMs and yields actionable privacy mitigation recommendations for model training, API access control, and decoding configurations.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: (Large) Language Models
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Indistinguishability properties such as differential privacy bounds or low empirically measured membership inference are widely treated as proxies to show a model is sufficiently protected against broader memorization risks. However, we show that indistinguishability properties are neither sufficient nor necessary for preventing data extraction in LLM APIs. We formalize a privacy-game separation between extraction and indistinguishability-based privacy, showing that indistinguishability and inextractability are incomparable: upper-bounding distinguishability does not upper-bound extractability. To address this gap, we introduce $(l, b)$-inextractability as a definition that requires at least $2^b$ expected queries for any black-box adversary to induce the LLM API to emit a protected $l$-gram substring. We instantiate this via a worst-case extraction game and derive a rank-based extraction risk upper bound for targeted exact extraction, as well as extensions to cover untargeted and approximate extraction. The resulting estimator captures the extraction risk over multiple attack trials and prefix adaptations. We show that it can provide a tight and efficient estimation for standard greedy extraction and an upper bound on the probabilistic extraction risk given any decoding configuration. We empirically evaluate extractability across different models, clarifying its connection to distinguishability, demonstrating its advantage over existing extraction risk estimators, and providing actionable mitigation guidelines across model training, API access, and decoding configurations in LLM API deployment. Our code is publicly available at: https://github.com/Emory-AIMS/Inextractability.

Problem

• data extraction
• LLM APIs
• indistinguishability
• privacy risk
• inextractability

Innovation

• inextractability
• privacy-game separation
• extraction risk estimation
• black-box LLM API
• membership inference