Search over the Visual World: Persistent Visual Memory, Layered Indexes, and Source-Grounded Evidence

2026-08-08

AI Summary

This work addresses the challenges of efficient retrieval, contextual selection, and evidential traceability in dynamic visual data streams by proposing a system-level abstraction that models visual memory as a multi-layered scene space aligned on a shared timeline. The design emphasizes source evidence traceability and decouples key decisions—such as model selection, segmentation, and embedding—into configurable components. Built upon the VideoDB format and a hierarchical indexing structure, the system treats real-time video streams as first-class data sources and offers typed search interfaces to support planned retrieval, stateful investigation, and evidence synthesis. Evaluated on four public datasets with over 9,800 queries, the generic component pipeline achieves Recall@1/@3/@10 of 73.09/83.39/91.20, significantly outperforming commercial video-native engines and demonstrating the critical impact of the proposed system architecture on retrieval performance.

Abstract

Most video-retrieval systems assume a bounded corpus and return ranked files or timestamps. Agents operating over cameras, screens, streams, and archives face a different systems problem: observations arrive continuously; models interpret them at different temporal granularities; context must be selected without replaying the complete visual record; and results must stay connected to inspectable source evidence. We argue that search over such a corpus is an infrastructure problem that cannot be reduced to ranking video files. We develop a conceptual and formal model of search over the visual world built on analyzer-defined scenes, persistent understanding artifacts, visual memory as coexisting scene spaces over shared source time, and capability-declared indexes, distinguishing memory (everything retained), context (what is selected for a task), and evidence (the source intervals that ground it). The VideoDB data format (VDB) realizes this model in production, exposed through a typed search surface spanning planned retrieval, stateful investigation, direct access, and grounded synthesis. We contrast this model-agnostic infrastructure, where segmentation, sampling, model choice, embeddings, and ranking are system decisions and live streams are first-class sources, with video-native foundation models offered as fixed APIs. In a semantic-retrieval comparison against a commercial video-native engine spanning 9,800+ queries over four public datasets, a pipeline of general-purpose components achieves higher macro-averaged Recall@1/@3/@10 (73.09/83.39/91.20 versus 65.75/77.13/89.10), while the baseline is higher at Recall@50 (96.42 versus 96.07). Retrieval quality over the visual world is today governed more by system design than by video-specific pretraining, and visual-memory infrastructure can deliver it while keeping playable, source-grounded evidence first-class.

Problem

• video retrieval
• visual memory
• continuous visual streams
• source-grounded evidence
• temporal granularity

Innovation

• persistent visual memory
• layered indexes
• source-grounded evidence
• VideoDB
• visual world search