🤖 AI Summary
Large language models are constrained by fixed context windows, leading to irreversible loss of dialogue history and degraded long-range coherence. This work proposes a purely application-layer dual-memory architecture that achieves near-infinite effective context without modifying the underlying model or introducing vendor lock-in. The architecture integrates episodic memory—based on sliding-window vector embeddings—with agent-writable structured semantic memory, and introduces, for the first time, an intra-conversational retrieval mechanism that enables efficient access to compressed dialogue history. A novel MOIM (Memory-Oriented Information Merging) adaptive injection mechanism dynamically incorporates relevant contextual information. The system supports zero-overhead fallback to native LLM behavior and significantly enhances contextual coherence in extended conversations.
📝 Abstract
Large language models (LLMs) operate within fixed context windows that fundamentally limit conversational continuity. When context fills, compaction discards history irreversibly; when sessions end, all memory resets to zero. Existing solutions-larger context windows, retrieval-augmented generation for knowledge bases, and memory-augmented architectures such as MemGPT-either require model modification, impose provider lock-in, or do not address the compaction continuity problem. We present CALMem (Conversational Application-Layer Memory), an application-layer dual memory architecture that gives LLM-based conversational assistants virtually unbounded effective context without any modification to the underlying model. CALMem combines two complementary memory subsystems: an episodic memory layer built on sliding-window vector embeddings of conversation history, and a semantic memory layer of agent-writable structured facts. A token-budget-adaptive injection mechanism, called the MOIM (Message of Injected Memory), automatically retrieves and injects relevant past context each turn, scaling injection depth inversely with context pressure. A key contribution is intra-session retrieval: compacted away turns from the current session remain searchable, closing a gap unaddressed by prior work. The system is implemented as a pure application layer in a production Rust codebase, is provider-agnostic, and degrades to original LLM behaviour with zero overhead when disabled. We describe the architecture, design decisions, and performance characteristics, and analyse the trade-offs that guided each implementation choice.