To Redact, or not to Redact? A Local LLM Approach to Deliberative Process Privilege Classification

2026-05-11

AI Summary

This study addresses the challenge of automatically identifying sensitive content protected under deliberative process privileges—such as Exemption 5 of the Freedom of Information Act (FOIA)—in government disclosures. The authors propose a localized classification approach that operates on consumer-grade hardware, leveraging the Qwen3.5-9B small language model enhanced with chain-of-thought reasoning and few-shot prompting that includes negative examples to enable sentence-level sensitivity detection without reliance on cloud-based APIs. Their analysis reveals that deliberative statements frequently combine first-person pronouns with opinion-expressing verbs, underscoring the importance of multi-faceted linguistic indicators. Experimental results demonstrate that the proposed method outperforms existing models in recall and F2 score, achieving performance comparable to the low-cost commercial model Gemini 2.5 Flash, thereby confirming the feasibility and practical utility of compact local models for identifying sensitive governmental information.

Abstract

Government transparency laws, like the Freedom of Information (FOIA) acts in the United States and United Kingdom, and the Woo (Open Government Act) in the Netherlands, grant citizens the right to directly request documents from the government. As these documents might contain sensitive information, such as personal information or threats to national security, the laws allow governments to redact sensitive parts of the documents prior to release. We build on prior research to perform automatic sensitivity classification for the FOIA Exemption 5 deliberative process privilege using Large Language Models (LLMs). However, processing documents not yet cleared for review via third-party cloud APIs is often legally or politically untenable. Therefore, in this work, we perform sensitivity classification with a small, local model, deployable on consumer-grade hardware (Qwen3.5 9B). We compare eight variants of applying LLMs for sentence classification, using well-known prompting techniques, and find that a combination of Chain-of-Thought prompting and few-shot prompting with error-based examples outperforms classification models of earlier work in terms of recall and F2 score. This method also closely approaches the performance of a widely-used, cost-efficient commercial model (Gemini 2.5 Flash). In an additional analysis, we find that sentences that are predicted as deliberative contain more verbs that indicate the expression of opinions, and are more often phrased in in first-person. Above all, deliberativeness seems characterized by the presence of a combination of multiple indicators, in particular the combination of first-person words with a verb for expressing opinion.

Problem

• deliberative process privilege
• redaction
• government transparency
• sensitivity classification
• FOIA

Innovation

• local LLM
• deliberative process privilege
• prompt engineering
• FOIA redaction
• sensitivity classification