SCOUT: Semantic Concept Discovery for Open-Vocabulary Editing of face Recognition Templates

2026-08-17

AI Summary

This study addresses the lack of direct semantic editing in face template spaces and the limitations of existing annotation-dependent methods with poor scalability. We propose an end-to-end framework leveraging mechanistic interpretability that learns sparse representations and generates natural language hypotheses to achieve open-vocabulary editing of face templates without re-encoding. Compatible with mainstream backbone networks, this approach transcends predefined attributes by discovering interpretable concepts beyond standard labels. It enables controllable manipulation with negligible identity loss while supporting independent decoding for visualization of edited templates. Consequently, this work significantly enhances both the semantic interpretability and operational flexibility of face template spaces, offering a robust solution for direct, label-free semantic control in facial representation learning.

Abstract

Face recognition templates are compact identity representations, yet they also encode rich semantic information about facial appearance. Prior work has shown that templates can be inverted to images or indirectly manipulated through image-editing pipelines, but direct semantic editing in template space remains largely unexplored. Existing interpretability methods for face recognition often rely on manual neuron inspection or predefined attribute labels, limiting scalability and semantic flexibility. To address this gap, we propose SCOUT (Semantic Concept Discovery for Open-VocabUlary Editing of Face Recognition Templates), an end-to-end framework for discovering and directly manipulating semantic concepts in face recognition templates using mechanistic interpretability. SCOUT learns sparse template representations, generates semantic hypotheses for latent features from natural-language descriptions, and validates their stability. The resulting features act as controllable semantic directions for direct editing, avoiding costly edit--re-encode pipelines. Experiments with face recognition models using CNN, ViT, and Swin backbones show that SCOUT discovers interpretable concepts beyond standard attribute labels and enables controllable, identity-aware template manipulation with negligible impact on identity matching. We further show that edited templates can subsequently be decoded with independent inversion models for visualization and evaluation.

Problem

• Face Recognition Templates
• Semantic Editing
• Mechanistic Interpretability
• Open-Vocabulary
• Concept Discovery

Innovation

• Mechanistic Interpretability
• Open-Vocabulary Editing
• Face Recognition Templates
• Sparse Representation
• Semantic Concept Discovery