Probing Privacy Leaks in LLM-based Code Generation via Test Generation

2026-05-14

AI Summary

This work addresses the risk of personally identifiable information (PII) leakage from large language models (LLMs) in code generation, a concern inadequately captured by existing detection methods that rely on manual or synthetic prompts and fail to reflect real-world usage. To overcome this limitation, the authors propose a test-driven privacy leakage detection paradigm that automatically generates realistic software test cases to elicit memorized PII from LLMs. Instead of conventional prompt engineering, their approach constructs an automated privacy signature database to enable contextual, black-box testing without human intervention. Empirical evaluation across five mainstream LLMs demonstrates that the proposed method identifies 2.56 times more confirmed PII leaks than current baselines, substantially improving both detection coverage and ecological validity.

Abstract

The widespread availability of large-scale code datasets has fueled the rapid development of large language models (LLMs) for code-related tasks. These datasets may include sensitive personally identifiable information (PII), which can lead to privacy leakage when LLMs memorize and reproduce it. However, existing privacy-leakage detection methods rely on ad-hoc prompt construction (manually or automatically designed). Therefore, they do not adequately approximate the real-world contexts in which PII appears in code corpora, making it difficult to extract realistic privacy leakage. In this paper, we propose a pipeline that simulates practical privacy-related code generation scenarios and adopts a test-driven strategy to elicit the memorized information from the generated test cases. We further introduce an automatically constructed privacy feature library that replaces manual prompt engineering by providing realistic templates and examples to guide test case generation. Large-scale experiments on 5 widely used LLMs show that our pipeline exposes more confirmed privacy leakage, achieving a 2.56 times increase in detected leakage compared to existing baselines.

Problem

• privacy leakage
• large language models
• code generation
• personally identifiable information
• test generation

Innovation

• privacy leakage detection
• test-driven generation
• privacy feature library
• LLM memorization
• code generation