MemeMind: Reference-Guided Trace Construction for Offline Context Optimization

2026-08-10

AI Summary

This work addresses the challenge in offline context optimization where only failed trajectories are available, particularly in complex multimodal tasks such as anime and gaming memes, which lack successful examples of tool usage. To overcome this limitation, the authors propose a reference-answer-guided trajectory reconstruction method. Their framework employs TraceBuilder to synthesize successful tool-invocation trajectories—encompassing text search, image retrieval, and visual grounding—by leveraging reference answers. ToolGuide then distills both shared guidelines and tool-specific instructions from these reconstructed trajectories to optimize reasoning contexts under frozen model settings. This approach is the first to exploit offline reference answers for reconstructing successful experiences, effectively complementing failed queries and yielding generalizable tool-use policies. On the MemeX benchmark, it boosts performance of Qwen3-VL-30B and 235B models by 22.0%/21.1% and 8.1%/8.0%, respectively (as evaluated by GPT-5), with ablation studies confirming that trajectory construction contributes the largest performance gain.

Abstract

Offline context optimization improves an agent by revising its instructions and examples while keeping the model frozen. This approach learns from rollouts on an adaptation set, but some queries produce only failed rollouts. In these cases, the optimizer sees no successful example of how the available tools can reach the correct answer. We introduce MemeMind, which uses an offline reference answer to recover this missing experience. TraceBuilder identifies the evidence required by the reference, executes text search, image retrieval, and visual grounding, and verifies the resulting tool trace before adding it to the adaptation buffer. ToolGuide then summarizes the collected traces into a shared guide and separate instructions for each tool. The reference answers and constructed traces are used only during adaptation, while inference uses the learned guides with a frozen model. We study this problem through Anime, Comic, and Game meme interpretation. These memes combine edited and ambiguous visual content, overlaid text, long tail franchise knowledge, and culture specific references. Their interpretation can require coordinated visual grounding, image retrieval, and text search, making them a demanding setting in which native rollout groups may fail together. We evaluate MemeMind on MemeX, a benchmark of 1,000 such memes annotated by experts. Across two Qwen3-VL models, two language partitions, and two independent judges, MemeMind improves over the strongest context optimization baseline by 22.0% and 21.1% on Qwen3-VL-30B-A3B, and by 8.1% and 8.0% on Qwen3-VL-235B-A22B under GPT-5 judging. Ablations and held out traces show that constructing successful tool use for failed groups provides the largest component gain and produces more effective evidence acquisition at inference time.

Problem

• offline context optimization
• failed rollouts
• tool trace construction
• reference-guided learning
• memes interpretation

Innovation

• reference-guided trace construction
• offline context optimization
• tool trace verification
• frozen model adaptation
• multimodal reasoning