Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy

2026-08-31

AI Summary

研究通过引入ECCBench评估视觉-语言模型的记忆性能，不仅限于准确性，还考量效率、压缩性和校准性，发现非Transformer架构在某些方面优于RoPE Transformers。

Abstract

Memory is widely viewed as an important unsolved problem for LLMs and VLMs, and current benchmarks typically evaluate it by testing accuracy over long text or video. However, accuracy alone misses properties that matter for real long-horizon tasks. We introduce ECCBench, a benchmark and evaluation protocol that measures memory beyond a system's capacity--its raw accuracy at a specific budget--via three axes we call ECC: efficiency--the computation, in FLOPs, needed to answer from memory; compression--whether compressible inputs are remembered more accurately or efficiently; and calibration--whether the system abstains in response to its own uncertainty and the cost of an error. We find that pretrained VLMs compress their memory over text but not video and are poorly calibrated on both. Among a broader set of memory backbones, several non-Transformer architectures achieve better compression-calibration tradeoffs than RoPE Transformers, suggesting they may be useful components for agents operating over long horizons.

Problem

• memory
• accuracy
• long-horizon tasks

Innovation

• ECCBench
• memory evaluation
• compression
• calibration
• non-Transformer architectures