Triage: Hierarchical Visual Budgeting for Efficient Video Reasoning in Vision-Language Models

2026-01-30

AI Summary

This work addresses the inefficiency of vision-language models in video reasoning, where redundant visual data leads to excessively long token sequences and high computational costs. To tackle this, the authors propose Triage, a training-free, plug-and-play framework that introduces a two-level hierarchical visual budgeting mechanism to dynamically allocate computational resources at both the frame and token levels. By integrating keyframe selection, core token retention, and a batch-wise Maximal Marginal Relevance (MMR)-based strategy for contextual token pruning, Triage significantly accelerates inference and reduces memory consumption. Remarkably, it maintains or even surpasses the performance of existing methods across multiple video understanding benchmarks, demonstrating an effective balance between efficiency and accuracy.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Vision-Language Models (VLMs) face significant computational challenges in video processing due to massive data redundancy, which creates prohibitively long token sequences. To address this, we introduce Triage, a training-free, plug-and-play framework that reframes video reasoning as a resource allocation problem via hierarchical visual budgeting. Its first stage, Frame-Level Budgeting, identifies keyframes by evaluating their visual dynamics and relevance, generating a strategic prior based on their importance scores. Guided by this prior, the second stage, Token-Level Budgeting, allocates tokens in two phases: it first secures high-relevance Core Tokens, followed by diverse Context Tokens selected with an efficient batched Maximal Marginal Relevance (MMR) algorithm. Extensive experiments demonstrate that Triage improves inference speed and reduces memory footprint, while maintaining or surpassing the performance of baselines and other methods on various video reasoning benchmarks.

Problem

• video reasoning
• vision-language models
• computational efficiency
• token redundancy
• visual budgeting

Innovation

• hierarchical visual budgeting
• frame-level budgeting
• token-level budgeting
• Maximal Marginal Relevance
• training-free video reasoning