Can Large Language Models Really Recognize Your Name?

2025-05-20

AI Summary

This work exposes a systematic failure of large language models (LLMs) in detecting personally identifiable information (PII)—specifically, ambiguous personal names—under privacy-sensitive contexts, undermining their reliability as PII detection tools. To address this, the authors introduce AMBENCH, the first fine-grained benchmark grounded in name regularity deviations, and design controlled prompt injection and named-entity ambiguity test suites, evaluating performance across both PII detection and privacy-preserving summarization tasks. Experiments reveal that state-of-the-art LLMs and dedicated PII tools suffer 20–40% recall degradation on ambiguous names; benign prompt injection quadruples false-negative rates; and sensitive information omission in privacy summaries increases markedly. This study is the first to quantitatively characterize how contextual ambiguity critically impairs LLMs’ privacy capabilities, establishing a reproducible evaluation framework and a novel benchmark—thereby providing critical insights and methodological foundations for developing trustworthy privacy-enhancing technologies.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Humans and AI: AI for Accessibility

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) are increasingly being used to protect sensitive user data. However, current LLM-based privacy solutions assume that these models can reliably detect personally identifiable information (PII), particularly named entities. In this paper, we challenge that assumption by revealing systematic failures in LLM-based privacy tasks. Specifically, we show that modern LLMs regularly overlook human names even in short text snippets due to ambiguous contexts, which cause the names to be misinterpreted or mishandled. We propose AMBENCH, a benchmark dataset of seemingly ambiguous human names, leveraging the name regularity bias phenomenon, embedded within concise text snippets along with benign prompt injections. Our experiments on modern LLMs tasked to detect PII as well as specialized tools show that recall of ambiguous names drops by 20--40% compared to more recognizable names. Furthermore, ambiguous human names are four times more likely to be ignored in supposedly privacy-preserving summaries generated by LLMs when benign prompt injections are present. These findings highlight the underexplored risks of relying solely on LLMs to safeguard user privacy and underscore the need for a more systematic investigation into their privacy failure modes.

Problem

• LLMs fail to reliably detect ambiguous human names in text
• Ambiguous names are often misinterpreted due to context biases
• Current LLM-based privacy solutions show significant recall drops

Innovation

• Proposes AMBENCH benchmark for ambiguous names
• Exposes LLMs' 20-40% recall drop on ambiguous names
• Reveals LLMs ignore names in privacy summaries