Foundation Cures Personalization: Improving Personalized Models' Prompt Consistency via Hidden Foundation Knowledge

2024-11-22

AI Summary

Facial personalized generation faces a fundamental trade-off between identity fidelity and prompt consistency. To address this, we propose FreeCure—a training-free framework that introduces, for the first time, a dual-path inference paradigm to identify semantically salient attributes requiring enhancement. FreeCure incorporates a base-aware self-attention module and a latent-space inversion mechanism, enabling precise calibration of personalized models’ responses to original base-model prompts—without altering any pre-trained weights. The method is fully non-invasive, zero-shot, and compatible with mainstream state-of-the-art facial personalization models. Extensive experiments demonstrate that FreeCure improves attribute controllability by 23.6% on average while strictly preserving identity fidelity, thereby significantly enhancing prompt effectiveness and generation controllability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Facial personalization faces challenges to maintain identity fidelity without disrupting the foundation model's prompt consistency. The mainstream personalization models employ identity embedding to integrate identity information within the cross-attention mechanisms of UNet. However, our preliminary experimental findings reveal that identity embeddings compromise the effectiveness of other tokens in the prompt, thereby limiting high prompt consistency and controllability. Moreover, by deactivating identity embedding, personalization models still demonstrate the underlying foundation models' ability to control facial attributes precisely. It suggests that such foundation models' knowledge can be leveraged to extbf{cure} the ill-aligned prompt consistency of personalization models. Building upon these insights, we propose extbf{FreeCure}, a framework that improves the prompt consistency of personalization models with their latent foundation models' knowledge. First, by setting a dual inference paradigm with/without identity embedding, we identify attributes ( extit{e.g.}, hair, accessories, etc.) for enhancements. Second, we introduce a novel foundation-aware self-attention module, coupled with an inversion-based process to bring well-aligned attribute information to the personalization process. Our approach is extbf{training-free}, and can effectively enhance a wide array of facial attributes in a non-intrusive manner; and it can be seamlessly integrated into existing popular personalization models, without harming their well-trained modules. FreeCure has demonstrated significant improvements in prompt consistency across a diverse set of state-of-the-art facial personalization models while maintaining the integrity of original identity fidelity. The project page is available href{https://github.com/YIYANGCAI/freecure-project-page}{here}.

Problem

• Improving prompt consistency in facial personalization models
• Leveraging foundation models to enhance attribute control
• Maintaining identity fidelity without disrupting model consistency

Innovation

• Dual inference paradigm enhances attribute identification.
• Foundation-aware self-attention improves prompt consistency.
• Training-free integration maintains model integrity.