StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs

2026-08-13

AI Summary

Streaming vision-language models face a fundamental trade-off between real-time perception and long-term memory: truncating context improves immediate comprehension but degrades the ability to reason over distant past events. This work proposes StreamTTT, the first streaming VLM architecture to decouple these mechanisms by employing online-updated fast weights to retain long-range history while preserving a sliding key-value cache for recent inputs, thereby mitigating attention dilution. Trained jointly on offline long-video QA data and a newly introduced real-time QA benchmark, StreamTTT-4B outperforms SimpleStream-4B by 1.4 and 3.7 points on real-time perception and retrospective reasoning tasks, respectively, on OVO-Bench, and matches the performance of the larger SimpleStream-8B on the RTVU subset of StreamingBench.

Abstract

Humans effortlessly perceive the present while remembering the past, yet streaming VLMs often trade off real-time perception against long-term memory. Prior work shows that shortening the context can sharpen current-scene perception at the expense of long-range recall. To reconcile these abilities, we introduce StreamTTT, which writes long-range history into online-updated fast weights outside the attention context. This leaves a short sliding key-value cache dedicated to recent evidence, mitigating attention dilution. We train StreamTTT jointly on offline long-video QA and a newly constructed real-time QA corpus. On OVO-Bench, StreamTTT-4B outperforms SimpleStream-4B by 1.4 points in real-time perception and 3.7 points in backward tracing. It also remains competitive with the larger SimpleStream-8B on the Real-Time Visual Understanding (RTVU) subset of StreamingBench. Our code will be released.

Problem

• streaming VLMs
• real-time perception
• long-term memory
• attention dilution
• visual language models

Innovation

• Streaming VLMs
• Fast Weights
• Real-Time Perception
• Long-Term Memory
• Attention Dilution