Online-PVLM: Advancing Personalized VLMs with Online Concept Learning

2025-11-25

AI Summary

Existing personalized vision-language models (VLMs) require separate embedding learning for each new concept, hindering real-time test-time adaptation and scalable, efficient retrieval. Method: We propose the first online concept learning framework for personalized VLMs, leveraging hyperbolic space to model semantic hierarchies and enabling zero-shot concept embedding generation; we further introduce OP-Eval—a dynamically updatable, large-scale evaluation benchmark covering cross-domain retrieval and diverse question-answering tasks. Contribution/Results: Experiments demonstrate significant improvements over baselines on OP-Eval, with millisecond-level concept insertion and retrieval. The framework achieves high efficiency, scalability, and practicality, establishing a novel paradigm for deploying personalized VLMs in real-world applications.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Personalized Visual Language Models (VLMs) are gaining increasing attention for their formidable ability in user-specific concepts aligned interactions (e.g., identifying a user's bike). Existing methods typically require the learning of separate embeddings for each new concept, which fails to support real-time adaptation during testing. This limitation becomes particularly pronounced in large-scale scenarios, where efficient retrieval of concept embeddings is not achievable. To alleviate this gap, we propose Online-PVLM, a framework for online concept learning by leveraging hyperbolic representations. Our approach makes a train-free paradigm for concept embeddings generation at test time, making the use of personalized VLMs both scalable and efficient. In addition, we develop OP-Eval, a comprehensive and large-scale benchmark comprising 1,292 concepts and over 30K high-quality instances with diverse question types, designed to rigorously assess online concept learning in realistic scenarios. Extensive experiments demonstrate the state-of-the-art performance of our proposed framework. Our source code and dataset will be made available.

Problem

• Enables real-time concept learning during testing phase
• Eliminates need for separate embedding training per concept
• Solves scalability issues in large-scale personalized VLM deployment

Innovation

• Online concept learning with hyperbolic representations
• Train-free paradigm for embedding generation
• Scalable and efficient personalized VLM framework