MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration

2026-04-16

AI Summary

Chain-of-thought (CoT) reasoning faces efficiency bottlenecks due to the linear growth of key-value (KV) cache with the number of generated tokens. This work proposes MemoSight, a novel framework that unifies context compression and multi-token parallel prediction within a single architecture for the first time. Through carefully designed special tokens, customized positional encodings, and efficient KV cache management, MemoSight holistically optimizes the inference process. Evaluated on four reasoning benchmarks, MemoSight reduces KV cache usage by up to 66% and achieves a 1.56× speedup while maintaining or even surpassing the reasoning performance of existing CoT methods.

Technology Category

• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

While Chain-of-thought (CoT) reasoning enables LLMs to solve challenging reasoning problems, as KV cache grows linearly with the number of generated tokens, CoT reasoning faces scaling issues in terms of speed and memory usage. In this work, we propose MemoSight (Memory-Foresight-based reasoning), a unified framework that integrates both context compression and multi-token prediction to mitigate the efficiency issues while maintaining CoT reasoning performance. Our framework adopts the same minimalist design for both context compression and multi-token prediction via special tokens and their corresponding position layout tailored to each token type. Comprehensive experiments on four reasoning benchmarks demonstrate that MemoSight reduces the KV cache footprint by up to 66% and accelerates inference by 1.56x, while outperforming existing CoT compression methods.

Problem

• Chain-of-thought reasoning
• KV cache
• reasoning acceleration
• context compression
• multi-token prediction

Innovation

• context compression
• multi-token prediction
• KV cache optimization
• Chain-of-thought reasoning
• efficient inference