Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models

2025-12-16

AI Summary

To address excessive computational and memory overhead in real-time inference of video-level vision-language models (VLMs), this paper proposes Focus—a streaming-focused architecture. Its core is a novel multi-level streaming concentration compression paradigm, comprising three tiers of fine-grained redundancy elimination: semantic-guided, prompt-aware token pruning; local spatiotemporal block compression; and motion-aware vector matching. Focus employs full-stack co-design—including convolutional memory layout, cross-modal attention optimization, an on-chip streaming token processing unit, and hardware-native support for modular systolic arrays—to enable low-latency, energy-efficient on-chip streaming execution. Evaluated on real hardware, Focus achieves 2.4× speedup and 3.3× energy reduction over state-of-the-art accelerators. The source code and complete full-stack implementation are open-sourced.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-Language Models (VLMs) have demonstrated strong performance on tasks such as video captioning and visual question answering. However, their growing scale and video-level inputs lead to significant computational and memory overhead, posing challenges for real-time deployment on hardware accelerators. While prior work attempts to reduce redundancy via token pruning or merging, these methods typically operate at coarse granularity and incur high runtime overhead due to global token-level operations. In this study, we propose Focus, a Streaming Concentration Architecture that efficiently accelerates VLM inference through progressive, fine-grained redundancy elimination. Focus introduces a multilevel concentration paradigm that hierarchically compresses vision-language inputs at three levels: (1) semantic-guided token pruning based on textual prompts, (2) spatial-temporal block-level concentration using localized comparisons, and (3) vector-level redundancy removal via motion-aware matching. All concentration steps are tightly co-designed with the architecture to support streaming-friendly, on-chip execution. Focus leverages GEMM tiling, convolution-style layout, and cross-modal attention to minimize off-chip access while enabling high throughput. Implemented as a modular unit within a systolic-array accelerator, Focus achieves a 2.4x speedup and 3.3x reduction in energy, significantly outperforming state-of-the-art accelerators in both performance and energy efficiency. Full-stack implementation of Focus is open-sourced at https://github.com/dubcyfor3/Focus.

Problem

• Reduces computational and memory overhead in vision-language models
• Eliminates fine-grained redundancy for efficient real-time inference
• Accelerates VLM deployment on hardware with streaming-friendly architecture

Innovation

• Hierarchical compression with semantic-guided token pruning
• Spatial-temporal block-level concentration via localized comparisons
• Streaming-friendly on-chip execution with GEMM tiling and cross-modal attention