Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing

2026-07-09

AI Summary

This work addresses the limitations of existing unified multimodal models in long-range, multi-turn visual dialogue—specifically, visual token inflation and unreliable cross-turn reference. The authors propose the Cognitive Structured Multimodal Agent (CMA), which externalizes visual information into episodic visual memory and selectively activates relevant memory segments during inference to enable efficient understanding, generation, and editing. CMA integrates a modular cognitive architecture, a programmable scene engine that synthesizes multi-turn data with fine-grained retrieval annotations, and the CMA-Harness deployment framework comprising perceptual abstraction, cognitive retrieval, and a multimodal execution controller. Experiments demonstrate that an 8B-parameter CMA achieves a 91.4% retrieval accuracy over 20 dialogue turns, outperforming a 32B baseline by 8.2%, while reducing per-turn inference latency from 23.1 to 12.7 seconds.

Abstract

Recent unified multimodal models show a single architecture can jointly perform vision/language understanding and image generation/editing. However, they repeatedly feed all historical visual and textual inputs into a shared context window, limiting long-horizon multimodal dialogue due to visual token explosion and unreliable cross-turn referencing. We propose a Cognitive-structured Multimodal Agent that externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning. The agent consists of a Perceptual Abstraction Engine for structured visual abstraction, a Cognitive Retrieval Engine for cross-turn memory retrieval, and a Multimodal Executive Controller for autonomous task inference and action planning. To address the lack of turn-level retrieval supervision in existing datasets, we develop a Unified Scenario Engine that programmatically generates structured multi-turn conversations with fine-grained retrieval annotations, enabling reinforcement learning to optimize abstraction and retrieval policies. We also construct a long-horizon visual-dialogue benchmark stratified by difficulty to evaluate episodic visual recall. Our 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving per-turn inference time (23.1s -> 12.7s). We further present the Cognitive-structured Multimodal Agent Harness (CMA-Harness), a tool-augmented deployment of the same cognitive structure integrating persistent multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving. Structured memory and modular decision-making offer a more scalable, efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling. Code: https://github.com/caseclose/cma-harness ; Project page: https://caseclose.github.io/cma-harness/

Problem

• multimodal dialogue
• visual token explosion
• cross-turn referencing
• long-horizon interaction
• episodic visual memory

Innovation

• Episodic Visual Memory
• Cognitive Retrieval
• Perceptual Abstraction
• Multimodal Agent
• Long-horizon Dialogue