MONKEY: Masking ON KEY-Value Activation Adapter for Personalization

2025-10-08

AI Summary

Diffusion models for personalized image generation often neglect text prompts, leading to insufficient semantic alignment. This paper proposes Mask-IPAdapter, a plug-and-play inference-time method that automatically generates subject masks via IP-Adapter and, in a secondary forward pass, masks out background image tokens—thereby directing text prompt attention to non-subject regions. Furthermore, it introduces a mask-guided KV activation adaptation mechanism enabling dual-path feature modulation during inference. Crucially, Mask-IPAdapter requires no fine-tuning or additional training, introducing only lightweight mask operations at inference time. Experiments on multiple personalized generation benchmarks demonstrate substantial improvements: +23.6% in CLIP-Score (prompt alignment) and +18.4% in ID-Sim (subject fidelity), with particularly strong gains in complex scene description tasks. The approach establishes a new paradigm for efficient, controllable, text-driven personalized image generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Personalizing diffusion models allows users to generate new images that incorporate a given subject, allowing more control than a text prompt. These models often suffer somewhat when they end up just recreating the subject image, and ignoring the text prompt. We observe that one popular method for personalization, the IP-Adapter automatically generates masks that we definitively segment the subject from the background during inference. We propose to use this automatically generated mask on a second pass to mask the image tokens, thus restricting them to the subject, not the background, allowing the text prompt to attend to the rest of the image. For text prompts describing locations and places, this produces images that accurately depict the subject while definitively matching the prompt. We compare our method to a few other test time personalization methods, and find our method displays high prompt and source image alignment.

Problem

• Prevents overfitting to subject images in personalization
• Uses automatic masks to separate subject from background
• Enhances text prompt alignment while preserving subject accuracy

Innovation

• Uses automatically generated masks for subject segmentation
• Applies masks to restrict image tokens to subject
• Enhances text prompt alignment with background context