Differentially Private Contrastive Learning via Bounding Group-level Contribution

2026-04-29

AI Summary

This work addresses the significant utility degradation in differentially private contrastive learning caused by strong inter-sample dependencies. To mitigate this issue, the authors propose a structured sensitivity-reduction framework that limits gradient coupling among samples within a batch by introducing a novel group contribution clipping mechanism, effectively controlling gradient sensitivity while preserving privacy guarantees. Coupled with an intra-group data augmentation strategy, the method enhances negative sample diversity without incurring additional privacy costs. Extensive experiments demonstrate that the proposed approach substantially outperforms existing methods across eight benchmark datasets, achieving a 5.6% improvement in image classification accuracy and a 20.1% gain in image-text retrieval accuracy.

Abstract

Differentially private (DP) contrastive learning aims to learn general-purpose representations from sensitive data, alleviating the privacy leakage concerns of organizations deploying or sharing embedding models trained on private user content. However, existing approaches suffer from severe utility degradation due to the over-strong inter-sample dependency inherent in standard contrastive objectives, where each sample's gradient depends on all other samples in the batch, amplifying the impact of DP noise. In this work, we argue that effective DP contrastive learning requires explicitly reducing such intrinsic inter-sample reliance. To this end, we propose DP-GCL, a principled DP contrastive learning framework that structurally limits gradient dependency through bounding group-level contribution. DP-GCL partitions each batch into small, disjoint groups and restricts available negative samples to within-group samples, thereby localizing gradient influence and reducing sensitivity. To counteract the resulting loss of negative sample diversity, we further introduce intra-group augmentation, which generates additional negative views without increasing privacy cost. Extensive experiments across eight datasets demonstrate that DP-GCL consistently advances the state of the art in both uni-modal and multi-modal contrastive learning under practical privacy budgets: it improves image classification accuracy by 5.6% and image-text retrieval accuracy by 20.1% over existing DP contrastive methods.

Problem

• Differentially Private
• Contrastive Learning
• Inter-sample Dependency
• Utility Degradation
• Privacy-Preserving Representation Learning

Innovation

• Differentially Private Learning
• Contrastive Learning
• Group-level Contribution
• Intra-group Augmentation
• Gradient Dependency