🤖 AI Summary
This work addresses four core challenges in privacy policy analysis: inefficient extraction of data collection statements, insufficient cross-policy pattern mining, difficulty in assessing terminological compliance and internal consistency, and misalignment between policy text and actual network traffic. To this end, we propose PoliGraph—the first integrated knowledge graph framework designed for full-policy analysis. Our approach features: (1) a local–global dual-layer ontology aligned with policy context, domain semantics, and legal requirements; (2) a hybrid extraction pipeline combining linguistics-driven NLP (PoliGrapher) and LLM-based prompt engineering (PoliGrapher-LM) to model semantic relationships at the statement level; and (3) built-in capabilities for inconsistency detection, terminological self-consistency evaluation, and policy–traffic alignment analysis. Evaluated on public benchmarks, PoliGraph achieves a 40% F1-score improvement over SOTA for data collection statement identification (97% precision), and significantly outperforms prior methods in contradiction detection and traffic-policy matching accuracy.
📝 Abstract
Privacy policies disclose how an organization collects and handles personal information. Recent work has made progress in leveraging natural language processing (NLP) to automate privacy policy analysis and extract data collection statements from different sentences, considered in isolation from each other. In this paper, we view and analyze, for the first time, the entire text of a privacy policy in an integrated way. In terms of methodology: (1) we define PoliGraph, a type of knowledge graph that captures statements in a policy as relations between different parts of the text; and (2) we revisit the notion of ontologies, previously defined in heuristic ways, to capture subsumption relations between terms. We make a clear distinction between local and global ontologies to capture the context of individual policies, application domains, and privacy laws. We develop PoliGrapher, an NLP tool to automatically extract PoliGraph from the text using linguistic analysis. Using a public dataset for evaluation, we show that PoliGrapher identifies 40% more collection statements than prior state-of-the-art, with 97% precision. In terms of applications, PoliGraph enables automated analysis of a corpus of policies and allows us to: (1) reveal common patterns in the texts across different policies, and (2) assess the correctness of the terms as defined within a policy. We also apply PoliGraph to: (3) detect contradictions in a policy, where we show false alarms by prior work, and (4) analyze the consistency of policies and network traffic, where we identify significantly more clear disclosures than prior work. Finally, leveraging the capabilities of the emerging large language models (LLMs), we also present PoliGrapher-LM, a tool that uses LLM prompting instead of NLP linguistic analysis, to extract PoliGraph from the policy text, and we show that it further improves coverage.
