Inverting Black-Box Face Recognition Systems via Zero-Order Optimization in Eigenface Space

2025-06-11

AI Summary

This work addresses privacy threats posed by black-box face recognition systems that output only similarity scores—without exposing internal embeddings. Method: We propose a novel face image reconstruction approach that operates solely on similarity feedback, eliminating the need for embedding access. Our method uniquely integrates zeroth-order optimization with PCA-based eigenface space constraints to achieve efficient, high-fidelity color face reconstruction under strict similarity-only conditions. Contribution/Results: By avoiding reliance on model internals (e.g., feature vectors), our approach significantly reduces query complexity. Extensive evaluation on LFW, AgeDB-30, and CFP-FP demonstrates state-of-the-art reconstruction accuracy under similarity-only settings, while achieving superior query efficiency compared to existing inversion attacks. This constitutes the first successful application of zeroth-order optimization with eigenface priors for black-box face reconstruction, establishing a new benchmark in privacy threat modeling against similarity-based recognition APIs.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Privacy
• Search and Optimization: Local Search

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Reconstructing facial images from black-box recognition models poses a significant privacy threat. While many methods require access to embeddings, we address the more challenging scenario of model inversion using only similarity scores. This paper introduces DarkerBB, a novel approach that reconstructs color faces by performing zero-order optimization within a PCA-derived eigenface space. Despite this highly limited information, experiments on LFW, AgeDB-30, and CFP-FP benchmarks demonstrate that DarkerBB achieves state-of-the-art verification accuracies in the similarity-only setting, with competitive query efficiency.

Problem

• Reconstruct faces from black-box models using similarity scores
• Address privacy threats via model inversion without embeddings
• Optimize in eigenface space for color face reconstruction

Innovation

• Zero-order optimization in eigenface space
• Reconstructs faces using only similarity scores
• PCA-derived eigenface space for color faces