Personal Visual Context Learning in Large Multimodal Models

2026-05-11

AI Summary

This work addresses the challenge of large models struggling to understand and leverage personalized visual context on wearable devices. To this end, we propose the Personal Visual Context Learning (Personal VCL) framework, which formalizes the task of personal visual context learning for the first time. We introduce Personal-VCL-Bench, a benchmark encompassing people, objects, and activities, and design an Agentic Context Bank architecture featuring a self-optimizing mechanism. This architecture dynamically integrates relevant visual information during inference through a user-specific visual memory bank, query-adaptive evidence retrieval, and multi-observation fusion. Experiments demonstrate that our approach significantly outperforms standard contextual prompting methods across diverse tasks and backbone models, thereby advancing the development of personalized multimodal assistants.

Abstract

As wearable devices like smart glasses integrate Large Multimodal Models (LMMs) into the continuous first-person visual streams of individual users, the evolution of these models into true personal assistants hinges on visual personalization: the ability to reason over visual information unique to the wearer. We formalize this capability as Personal Visual Context Learning (Personal VCL), the prompt-time capability of using user-specific visual context to resolve personalized queries. To systematically evaluate this, we present Personal-VCL-Bench, a comprehensive benchmark capturing the personal visual world across persons, objects, and behaviors. Our analysis of frontier LMMs identifies a profound context utilization gap, revealing that the mechanisms for leveraging visual evidence, as well as aggregating multiple visual observations, remain critically understudied. Motivated by these findings, we propose the Agentic Context Bank, a strong inference-time baseline that structures a user's visual context into a self-refining memory bank and employs query-adaptive evidence selection. Our baseline approach consistently improves over standard context prompting regimes across tasks and evaluated backbones, demonstrating a practical path towards future personalized LMMs.

Problem

• Personal Visual Context Learning
• Large Multimodal Models
• Visual Personalization
• First-person Vision
• Context Utilization

Innovation

• Personal Visual Context Learning
• Large Multimodal Models
• Agentic Context Bank
• Visual Personalization
• First-Person Vision