Privacy-Preserving Data Drift Detection and Recovery for Large-Scale LLM Applications via Proxy Representations

2026-08-08

AI Summary

This work addresses the challenge of detecting data drift and constructing representative evaluation sets in large-scale LLM deployments, where privacy constraints prohibit direct access to user interaction data. To overcome this limitation, the authors propose ProxyDrift, a framework that leverages multidimensional, non-PII proxy representations generated by LLMs to enable privacy-preserving drift detection and evaluation set updates. The approach introduces several innovations, including an opportunistic calibration of redundancy-aware alignment scores, a conditional proxy sampler, and round-trip consistency analysis, further enhanced by mutual information metrics linked to user satisfaction modeling. Evaluated in a billion-user setting, ProxyDrift achieves high alignment (RA ≈ 0.9), renders synthetic queries indistinguishable from real ones, and supports continuous drift monitoring alongside targeted data generation.

Abstract

LLM applications deployed at scale face a fundamental challenge: privacy constraints prevent direct inspection of user interactions, making it difficult to obtain any representative evaluation dataset or to track the ongoing evolution of production traffic. We present ProxyDrift, a framework that (i) identifies and measures drift between production traffic and offline evaluation sets, and (ii) constructs and refreshes those evaluation sets accordingly; all without access to raw user data. Our approach operates entirely on non-PII proxy representations: structured, multi-dimensional descriptors derived from LLM-based classification of user interactions. We introduce (1) a chance-calibrated, redundancy-aware (RA) alignment score that aggregates per-dimension drift measurements via mutual information; (2) a conditional sampler that generates synthetic proxies respecting inter-dimensional dependencies; (3) a roundtrip consistency analysis that exposes generator/classifier disagreements and guides proxy taxonomy refinement; and (4) a feedback-linkage analysis that ties per-dimension and per-value proxy distributions to user satisfaction, surfacing actionable failure and success modes. Serving hundreds of millions of users, ProxyDrift enables continuous drift monitoring and targeted synthetic data generation without exposing sensitive user data. Experiments confirm strong roundtrip consistency, discriminator-level indistinguishability of synthetic queries from human queries, and tight end-to-end alignment (RA~0.9) with production.

Problem

• privacy-preserving
• data drift
• large-scale LLM applications
• evaluation dataset
• production traffic

Innovation

• privacy-preserving
• data drift detection
• proxy representations
• synthetic data generation
• mutual information