Memory-Conditioned Tool Calling for Camera-First Visual Agents

2026-07-10

AI Summary

This work addresses the challenge of precise tool invocation by vision-based agents in camera-first settings, where only image inputs are available and user intent must be inferred accurately. The authors propose a memory-driven user alignment approach that constructs a three-layer structured visual memory—comprising user profiles, short-term focus, and observation logs—to condition large language model–based tool selection at each interaction step. A conflict-aware memory write-back mechanism is further introduced to dynamically refine the user model. This is the first method to integrate structured personal visual memory into the decision-making pipeline of camera-first agents. Experiments on an 800-image test set demonstrate that the full memory module improves tool-query relevance by 0.47 points on a 5-point scale (+11.2%) and enhances end-to-end task utility by 9.7%.

Abstract

Recognition tells an agent what is in an image; personal memory affects what is worth looking up next. In a camera-first setting the user can send only an image, so the agent must form the lookups. We study whether personal visual memory improves agent-side tool choice and tool arguments, and thereby more user-aligned multi-tool lookups. The design uses a three-layer personal visual memory (profile, short-term focus, observations) that is loaded on each turn to condition an LLM tool-calling loop under camera-first intake, and includes conflict-aware write-back intended to refresh the user model for later captures. On 800 images paired with synthetic memory blocks constructed for controlled ablation, removing the full three-layer memory block reduces tool-query relevance by 0.47 points absolute (4.21 -> 3.74 on a 5-point scale; 11.2% relative) and end-to-end utility by 0.082 absolute (0.842 -> 0.760; 9.7% relative). These results measure memory conditioning of tool policy under image-only intake with fixed synthetic blocks, not multi-session write-back from live user histories.

Problem

• visual agents
• tool calling
• personal memory
• camera-first
• memory conditioning

Innovation

• memory-conditioned tool calling
• camera-first visual agents
• personal visual memory
• conflict-aware write-back
• LLM tool selection