Rethinking Benchmarks for Differentially Private Image Classification

2026-01-23

AI Summary

Existing benchmarks for differentially private image classification lack diversity and systematic design, making it difficult to comprehensively evaluate method performance across varied scenarios. To address this gap, this work proposes the first standardized benchmark framework that encompasses multiple practical settings—including with and without auxiliary data, convex optimization regimes, and a range of heterogeneous datasets—and integrates classic differentially private training methods such as DP-SGD for systematic evaluation. The project also establishes a public leaderboard to foster reproducible and comparable research within the community. This benchmark reveals significant performance disparities of current techniques under different configurations, thereby providing a reliable reference and a unified evaluation platform for future research in differentially private machine learning.

Technology Category

• Machine Learning: Privacy
• Computer Vision: Bias, Fairness & Privacy
• Search and Optimization: Non-convex Optimization

Application Category

• Security and Privacy: Security and privacy of machine learning and AI applications
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

We revisit benchmarks for differentially private image classification. We suggest a comprehensive set of benchmarks, allowing researchers to evaluate techniques for differentially private machine learning in a variety of settings, including with and without additional data, in convex settings, and on a variety of qualitatively different datasets. We further test established techniques on these benchmarks in order to see which ideas remain effective in different settings. Finally, we create a publicly available leader board for the community to track progress in differentially private machine learning.

Problem

• differentially private
• image classification
• benchmarks
• machine learning
• evaluation

Innovation

• differentially private image classification
• comprehensive benchmarks
• leaderboard
• convex settings
• privacy-preserving machine learning