Ego: Embedding-Guided Personalization of Vision-Language Models

2026-03-10

AI Summary

Existing vision-language models often rely on additional training or external modules for personalization, which limits their generalization, scalability, and deployment efficiency. This work proposes a lightweight, fine-tuning-free approach that leverages the model’s intrinsic attention mechanisms to automatically extract visual tokens representing target concepts as “concept memories.” During inference, these memories enable efficient personalized responses through embedding guidance. The method unifies support for single-concept, multi-concept, and video-based scenarios, consistently outperforming current state-of-the-art approaches across diverse settings. Notably, it achieves high performance with minimal computational overhead, demonstrating strong generality and practical utility.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

AI assistants that support humans in daily life are becoming increasingly feasible, driven by the rapid advancements in multimodal language models. A key challenge lies in overcoming the generic nature of these models to deliver personalized experiences. Existing approaches to personalizing large vision language models often rely on additional training stages, which limit generality and scalability, or on engineered pipelines with external pre-trained modules, which hinder deployment efficiency. In this work, we propose an efficient personalization method that leverages the model's inherent ability to capture personalized concepts. Specifically, we extract visual tokens that predominantly represent the target concept by utilizing the model's internal attention mechanisms. These tokens serve as a memory of that specific concept, enabling the model to recall and describe it when it appears in test images. We conduct a comprehensive and unified evaluation of our approach and SOTA methods across various personalization settings including single-concept, multi-concept, and video personalization, demonstrating strong performance gains with minimal personalization overhead.

Problem

• personalization
• vision-language models
• embedding
• multimodal AI
• concept representation

Innovation

• personalization
• vision-language models
• attention mechanisms
• visual tokens
• embedding-guided