Dynamic Context Adapters: Efficiently Infusing History into Vision-and-Language Models

2026-08-11

AI Summary

This work addresses the challenge of efficiently integrating historical context in sequential decision-making tasks with vision-language models, where naively concatenating past frames drastically increases attention computation complexity and memory consumption. To overcome this limitation, the authors propose a Dynamic Context Adapter (DCA)—a lightweight, fixed-size dynamic memory compression mechanism that injects temporal information without explicitly concatenating historical frames, while preserving the original architecture of pretrained vision-language models. By effectively balancing memory capacity and computational efficiency, DCA achieves significant performance gains in long-horizon tasks, reduces attention computation by over 25%, and decreases memory usage by 13%.

Abstract

Historical context integration presents a fundamental challenge for Vision-Language Models (VLMs) in sequential decision-making tasks. Current VLMs process visual inputs independently, which creates critical limitations for downstream applications that require temporal understanding. Direct incorporation of historical frames into Transformer inputs produces quadratic attention complexity and excessive memory consumption. Existing approaches suffer from significant drawbacks: computational inflation or substantial information loss through temporal compression. To address these challenges, we introduce Dynamic Context Adapter (DCA), a novel context injection approach for pretrained VLMs. Our method employs fixed-size, dynamically compressed memory to preserve historical semantics without frame concatenation. DCA bridges static VLMs and recurrent policies and enables memory capabilities in pretrained models while maintaining computational efficiency. DCA achieves over $25\%$ reduction in attention FLOPs and $13\%$ memory savings while improving performance on long-horizon tasks.

Problem

• Vision-Language Models
• Historical Context Integration
• Temporal Understanding
• Computational Efficiency
• Memory Consumption

Innovation

• Dynamic Context Adapter
• Vision-Language Models
• Temporal Context
• Memory Compression
• Efficient Attention