Compressing Observation History into Agent Memory: Distilling Transformers into Recurrent Transformers

2026-06-19

AI Summary

This work addresses the challenge of deploying Transformers in long-sequence streaming vision and robotics tasks, where standard architectures suffer from prohibitive computational complexity and existing recurrent variants underperform due to inefficient history compression. To bridge this gap, the authors propose a bottleneck representation-based knowledge distillation framework: a teacher model explicitly compresses the full-history Transformer’s observations into a fixed-size memory representation, which then supervises the training of a recurrent student model. This approach enables the recurrent architecture to effectively emulate the global Transformer’s information compression mechanism, achieving linear time complexity while substantially narrowing the performance gap with full-history models—thereby overcoming the longstanding trade-off between memory efficiency and representational capacity.

Abstract

Transformers are AI's workhorse with strong performance in modeling sequential data, but their computational cost becomes prohibitive when processing long sequences. We target long-horizon streaming vision and robotics applications like map-free pose estimation, where it is particularly impractical to store and maintain a history of observations. Recurrent Transformers address this limitation by maintaining fixed-size memory but their performance lags behind that of transformers operating over the full observation history. We argue that this gap does not stem from architectural limitations, but from differences in how these models learn to compress past information. Without access to an observation history, recurrent models must explicitly decide what to retain in memory at each step, a significantly harder learning problem. In this work, we propose a distillation approach that transfers the compression strategy of a classical full-history transformer to a recurrent variant. We enable this by designing a teacher model that explicitly compresses its observation history into a fixed-size bottleneck representation. By directly supervising the student's memory with this bottleneck representation, we align the two compression mechanisms. We show that this approach allows to train a recurrent latent robotic memory with linear-time complexity while substantially narrowing the performance gap to full-history transformers.

Problem

• Transformers
• Recurrent Transformers
• Observation History Compression
• Long-horizon Streaming
• Agent Memory

Innovation

• Recurrent Transformers
• Knowledge Distillation
• Memory Compression
• Sequential Modeling
• Linear-Time Complexity