ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Control

2025-10-20

AI Summary

This work addresses the low synthetic text quality, severe statistical distortion, and poor controllability in differentially private (DP) text generation. We propose the hierarchical ACTG framework and an RL-enhanced ARL method, operating in two stages: DP-tabular-data-driven feature learning, followed by DP fine-tuning coupled with reinforcement learning (RL)-based conditional generation. To mitigate reward hacking, we introduce an SFT anchoring mechanism and explicitly model interpretable features for fine-grained control. Under strong DP guarantees (ε ≤ 2), our approach improves MAUVE by 20% over state-of-the-art DP text synthesis methods and enables attribute-level controllable generation. The core contribution lies in decoupling privacy preservation from generation control—achieving, for the first time, a unified text synthesis paradigm that jointly ensures feature interpretability, RL training stability, and strict DP constraints.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Privacy
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Security and Privacy: Privacy-enhancing technologies

Abstract

Generating high-quality synthetic text under differential privacy (DP) is critical for training and evaluating language models without compromising user privacy. Prior work on synthesizing DP datasets often fail to preserve key statistical attributes, suffer utility loss from the noise required by DP, and lack fine-grained control over generation. To address these challenges, we make two contributions. First, we introduce a hierarchical framework that decomposes DP synthetic text generation into two subtasks: feature learning and conditional text generation. This design explicitly incorporates learned features into the generation process and simplifies the end-to-end synthesis task. Through systematic ablations, we identify the most effective configuration: a rich tabular schema as feature, a DP tabular synthesizer, and a DP fine-tuned conditional generator, which we term ACTG (Attribute-Conditioned Text Generation). Second, we propose Anchored RL (ARL), a post-training method that improves the instruction-following ability of ACTG for conditional generation. ARL combines RL to boost control with an SFT anchor on best-of-$N$ data to prevent reward hacking. Together, these components form our end-to-end algorithm ACTG-ARL, which advances both the quality of DP synthetic text (+20% MAUVE over prior work) and the control of the conditional generator under strong privacy guarantees.

Problem

• Enhancing synthetic text quality under differential privacy constraints
• Preserving statistical attributes while ensuring user privacy protection
• Improving fine-grained control over differentially private text generation

Innovation

• Hierarchical framework decomposes DP text generation
• DP fine-tuned conditional generator with tabular features
• Anchored RL method enhances control via SFT anchor