PIA-Bench: Towards Automated Privacy Impact Assessment with Large Language Models

2026-09-11

AI Summary

本文提出PIA-Bench，利用大型语言模型自动化隐私影响评估，通过分析499份专家编写的PIA来解决执行PIA所需的高专业门槛问题。

Abstract

Privacy impact assessment (PIA) is a critical instrument for institutions to proactively identify privacy risks and develop mitigation strategies before system deployment. While mandated across regulatory and institutional contexts, executing PIA requires extensive privacy and technical expertise, posing a particular challenge for teams without access to such resources. Prior work shows the potential of leveraging large language models (LLMs) to assist practitioners' privacy decisions, but little is known about how accurately and reliably LLMs can automate PIA. To this end, we develop PIA-Bench, the first open benchmark for evaluating LLMs on real-world PIAs. We first audited 499 expert-authored PIAs published by US federal agencies and curated 73 structured PIAs, comprising a total of 451 privacy risk and 831 mitigation items, to evaluate LLMs' ability to assess privacy risks and propose mitigations of complex systems. Our results show that off-the-shelf LLMs produce meaningful assessments and identify avenues for future improvement. Finally, we call for improving domain-specific workflows for LLM agents, developing accountable LLM infrastructure, and designing new quality standards for PIAs.

Problem

• Privacy Impact Assessment
• Large Language Models
• Privacy Risks
• Mitigation Strategies

Innovation

• Large Language Models
• Privacy Impact Assessment
• Benchmarking
• Automated Evaluation