ArtFace: Towards Historical Portrait Face Identification via Model Adaptation

2025-08-28

AI Summary

Addressing domain shift, high intra-class variability, and scarce annotated data in historical portrait identification, this paper proposes a cross-domain adaptive method integrating foundation models with conventional face recognition networks. Specifically, we fine-tune multimodal foundation models (e.g., CLIP), design a style-robust embedding fusion mechanism, and perform art-image-specific training via transfer learning to mitigate representation bias induced by diverse artistic styles. Evaluated on multiple public art portrait datasets—including WikiArt-Person and RAFA—our approach consistently outperforms existing state-of-the-art methods, achieving an average 9.3% improvement in Top-1 identification accuracy. Experimental results demonstrate that foundation models, when carefully adapted, exhibit strong generalization capability and practical utility for recognizing non-photorealistic facial images.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Deep Neural Architectures and Foundation Models
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Identifying sitters in historical paintings is a key task for art historians, offering insight into their lives and how they chose to be seen. However, the process is often subjective and limited by the lack of data and stylistic variations. Automated facial recognition is capable of handling challenging conditions and can assist, but while traditional facial recognition models perform well on photographs, they struggle with paintings due to domain shift and high intra-class variation. Artistic factors such as style, skill, intent, and influence from other works further complicate recognition. In this work, we investigate the potential of foundation models to improve facial recognition in artworks. By fine-tuning foundation models and integrating their embeddings with those from conventional facial recognition networks, we demonstrate notable improvements over current state-of-the-art methods. Our results show that foundation models can bridge the gap where traditional methods are ineffective. Paper page at https://www.idiap.ch/paper/artface/

Problem

• Identifying sitters in historical portrait paintings
• Addressing domain shift and stylistic variations in artworks
• Improving facial recognition accuracy using foundation models

Innovation

• Fine-tuning foundation models for artworks
• Integrating embeddings from conventional networks
• Bridging domain gap with model adaptation