MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoning

2026-01-29

AI Summary

This work addresses the challenge of long-horizon agent reasoning under limited context windows, where conventional text-based memory methods inefficiently allocate tokens uniformly and struggle to preserve critical information. The authors propose MemOCR, a novel approach that introduces visual layout into memory compression for the first time: by converting structured rich-text elements—such as headings and highlights—into rendered images, it leverages spatial arrangement to adaptively allocate information density, prioritizing key evidence under strict context budgets. Integrating reinforcement learning with budget-aware training, the model maintains robustness across varying memory constraints. Experimental results demonstrate that MemOCR significantly outperforms strong textual baselines on both multi-hop and single-hop question answering benchmarks, achieving notably more efficient reasoning performance especially under extreme context compression.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Long-horizon agentic reasoning necessitates effectively compressing growing interaction histories into a limited context window. Most existing memory systems serialize history as text, where token-level cost is uniform and scales linearly with length, often spending scarce budget on low-value details. To this end, we introduce MemOCR, a multimodal memory agent that improves long-horizon reasoning under tight context budgets by allocating memory space with adaptive information density through visual layout. Concretely, MemOCR maintains a structured rich-text memory (e.g., headings, highlights) and renders it into an image that the agent consults for memory access, visually prioritizing crucial evidence while aggressively compressing auxiliary details. To ensure robustness across varying memory budgets, we train MemOCR with reinforcement learning under budget-aware objectives that expose the agent to diverse compression levels. Across long-context multi-hop and single-hop question-answering benchmarks, MemOCR outperforms strong text-based baselines and achieves more effective context utilization under extreme budgets.

Problem

• long-horizon reasoning
• memory compression
• context budget
• interaction history
• information density

Innovation

• MemOCR
• visual memory
• layout-aware compression
• multimodal reasoning
• budget-aware reinforcement learning