M2Note: Continual Evolution of Vision Language Models via Mistake Notebook Learning

2026-07-01

AI Summary

This work addresses the limitations of current vision-language models, which often suffer from hallucination, overlook critical visual cues, and exhibit poor generalization, while conventional fine-tuning approaches are costly and hinder continuous improvement. The authors propose M2Note, a novel training-free framework that enables continual self-evolution by maintaining a structured “error notebook.” Failed cases are converted into retrievable subject-instruction dual notes, which are dynamically retrieved during inference via multimodal retrieval-augmented generation (RAG) to prevent recurring errors. M2Note incorporates batch-level rollback validation and a Chain-of-Thought–compatible architecture, facilitating both autonomous evolution and cross-model transferability. Experiments demonstrate consistent and significant performance gains across six benchmarks, highlighting its high sample efficiency, low computational cost, and strong complementarity with existing reasoning methods.

Abstract

Vision Language Models (VLMs) have demonstrated remarkable capabilities in multimodal reasoning tasks, yet they still suffer from recurring failures, such as skipping key visual checks, misapplying domain rules, and hallucinating unsupported concepts. Most existing solutions rely on supervised fine-tuning (SFT) and reinforcement learning (RL), which are expensive to iterate and can be brittle under distribution shift. To this end, we propose Multimodal Mistake Notebook Learning (M2Note), a training-free continual evolution framework that externalizes learning into an editable memory. M2Note transforms failed trajectories into compact subject-guidance notes: the subject summarizes the underlying domain and concept, while the guidance provides actionable verification steps that can be reused in future inference. At test time, M2Note retrieves relevant notes via multimodal retrieval-augmented generation (RAG) and appends them to the model context, steering reasoning away from previously observed pitfalls. To stabilize continual evolution, we adopt batch-level post-verification with rollback, which commits notebook edits only if they improve performance on the same batch, reducing noisy updates and preventing regressions. M2Note supports both self-evolving, where the same VLM acts as solver and supervisor, and cross-model evolving, where a stronger supervisor guides a weaker solver, enabling capability transfer without weight updates. Experiments on six multimodal reasoning benchmarks show consistent improvements across domains and backbones, while achieving strong cost and sample efficiency and remaining complementary to Chain-of-Thought (CoT) prompting.

Problem

• Vision Language Models
• multimodal reasoning
• hallucination
• recurring failures
• distribution shift

Innovation

• Mistake Notebook Learning
• Retrieval-Augmented Generation
• Continual Evolution
• Training-Free Adaptation
• Multimodal Reasoning