NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQA

2024-11-06

arXiv.org

AI Summary

To address the challenge of simultaneously ensuring privacy preservation and communication efficiency in real-world invoice processing, this paper introduces the first federated learning benchmark platform for Document Visual Question Answering (DocVQA), integrating document analysis, federated learning, and differential privacy. We propose a dual-track framework: Track 1 employs FedAvg with gradient compression and adaptive noise injection to maintain model utility under low communication overhead; Track 2 pioneers end-to-end document-level differential privacy for DocVQA, delivering provable privacy guarantees via fine-tuning of multimodal generative language models and optimized privacy budget allocation. Our contributions include a reproducible benchmark, a standardized paradigm for organizing federated privacy challenges, and foundational guidance toward establishing dual-dimensional (model- and data-centric) privacy practices in document image analysis.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

The Privacy Preserving Federated Learning Document VQA (PFL-DocVQA) competition challenged the community to develop provably private and communication-efficient solutions in a federated setting for a real-life use case: invoice processing. The competition introduced a dataset of real invoice documents, along with associated questions and answers requiring information extraction and reasoning over the document images. Thereby, it brings together researchers and expertise from the document analysis, privacy, and federated learning communities. Participants fine-tuned a pre-trained, state-of-the-art Document Visual Question Answering model provided by the organizers for this new domain, mimicking a typical federated invoice processing setup. The base model is a multi-modal generative language model, and sensitive information could be exposed through either the visual or textual input modality. Participants proposed elegant solutions to reduce communication costs while maintaining a minimum utility threshold in track 1 and to protect all information from each document provider using differential privacy in track 2. The competition served as a new testbed for developing and testing private federated learning methods, simultaneously raising awareness about privacy within the document image analysis and recognition community. Ultimately, the competition analysis provides best practices and recommendations for successfully running privacy-focused federated learning challenges in the future.

Problem

• Develop private federated learning solutions for invoice processing
• Extract and reason over document images with privacy constraints
• Reduce communication costs while maintaining utility in federated learning

Innovation

• Fine-tuned pre-trained Document VQA model
• Reduced communication costs in federated learning
• Applied differential privacy for document protection