Mitigating Knowledge Discrepancies among Multiple Datasets for Task-agnostic Unified Face Alignment

2025-03-28

AI Summary

Existing methods struggle to integrate heterogeneous facial landmark annotations from multiple sources, resulting in limited per-dataset training samples and insufficient generalization robustness. To address this, we propose TUFA—a task-agnostic unified facial alignment framework. Its core contributions are: (1) a semantic-alignment embedding-guided mean-face anchoring mechanism that establishes an interpretable cross-dataset alignment plane; (2) structured prompt encoding mapping, enabling zero-shot localization of unseen landmark types; and (3) multi-dataset joint training coupled with knowledge distillation for efficient few-shot transfer. TUFA achieves significant accuracy improvements across seven standard benchmarks. It substantially boosts performance in few-shot settings and, for the first time, enables zero-shot generalization to landmark types entirely absent during training.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Despite the similar structures of human faces, existing face alignment methods cannot learn unified knowledge from multiple datasets with different landmark annotations. The limited training samples in a single dataset commonly result in fragile robustness in this field. To mitigate knowledge discrepancies among different datasets and train a task-agnostic unified face alignment (TUFA) framework, this paper presents a strategy to unify knowledge from multiple datasets. Specifically, we calculate a mean face shape for each dataset. To explicitly align these mean shapes on an interpretable plane based on their semantics, each shape is then incorporated with a group of semantic alignment embeddings. The 2D coordinates of these aligned shapes can be viewed as the anchors of the plane. By encoding them into structure prompts and further regressing the corresponding facial landmarks using image features, a mapping from the plane to the target faces is finally established, which unifies the learning target of different datasets. Consequently, multiple datasets can be utilized to boost the generalization ability of the model. The successful mitigation of discrepancies also enhances the efficiency of knowledge transferring to a novel dataset, significantly boosts the performance of few-shot face alignment. Additionally, the interpretable plane endows TUFA with a task-agnostic characteristic, enabling it to locate landmarks unseen during training in a zero-shot manner. Extensive experiments are carried on seven benchmarks and the results demonstrate an impressive improvement in face alignment brought by knowledge discrepancies mitigation.

Problem

• Unify face alignment knowledge from diverse landmark datasets
• Enhance model robustness with limited single-dataset samples
• Enable zero-shot landmark prediction via interpretable semantic alignment

Innovation

• Unify knowledge from multiple face datasets
• Semantic alignment embeddings for mean shapes
• Task-agnostic zero-shot landmark localization