LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs

2026-05-17

AI Summary

This work addresses the computational bottleneck in long-form video understanding caused by the exponential growth of visual tokens, which persists even after token compression due to the high overhead of conventional vision encoders. To overcome this, the authors propose LiteFrame, an efficient vision encoder that integrates Compressed Token Distillation (CTD) and Language Model Alignment (LMA). CTD enables a lightweight student encoder to directly learn spatiotemporally compressed representations from a teacher model, circumventing redundant computations, while LMA ensures effective alignment with large language models. Under identical computational budgets, LiteFrame reduces end-to-end latency by 35% compared to InternVL3-8B, supports processing eight times more video frames, and achieves consistent accuracy gains across multiple video understanding benchmarks, establishing a new Pareto frontier between latency and performance.

Abstract

The fundamental challenge in scaling Video Large Language Models (Video LLMs) to long-form video lies in managing the explosion of visual-token context length. Existing strategies predominantly focus on "post-hoc" token reduction -- reducing visual tokens after feature extraction to alleviate the LLM's computational overhead. While these methods effectively reduce the number of visual tokens, we observe that the primary latency bottleneck then shifts from the LLM to the expensive per-frame processing of the vision encoder. To address this, we introduce LiteFrame, a strong, yet highly efficient video encoder backbone for Video LLMs. To train LiteFrame, we propose Compressed Token Distillation (CTD), a novel training framework that teaches a compact student vision encoder to directly predict information-dense, spatio-temporally compressed representations produced by a large teacher vision model, effectively bypassing redundant computation. When coupled with further Language Model Adaptation (LMA), this approach results in a new latency-accuracy Pareto frontier -- compared with InternVL3-8B, LiteFrame provides a 35% reduction in end-to-end latency while processing 8$\times$ more frames and improves average video understanding accuracy across multiple benchmarks. Our results demonstrate a new potential path to unlocking longer-form video understanding under fixed compute budgets.

Problem

• Video LLMs
• vision encoder
• visual-token context length
• long-form video
• computational bottleneck

Innovation

• LiteFrame
• Compressed Token Distillation
• Vision Encoder
• Video LLMs
• Efficient Inference