HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning

2026-06-19

AI Summary

Existing vision-language models struggle to effectively integrate fine-grained perception with multi-step, high-order temporal reasoning in long video understanding. To address this challenge, this work proposes the Hierarchical Programmed Probing (HPP) framework, which decouples perception and reasoning for the first time. HPP employs information-density-aware hierarchical video segmentation and leverages a code-generating large language model within an interactive programming environment to plan multi-step probing strategies, invoking vision-language models for localized perception on demand. By combining late-interaction semantic retrieval with structured probing functions, HPP enables coarse-to-fine temporal localization. The framework achieves significant performance gains across multiple benchmarks—including LongVideoBench, EgoSchema, VideoMME, and MLVU—demonstrating that decoupling perception from reasoning effectively supports both fine-grained understanding and long-range temporal inference.

Abstract

Understanding long videos requires fine-grained perception and multi-step, higher-order reasoning over complex, long-range spatio-temporal dynamics. Vision-language models (VLMs) encode video frames into visual tokens and attempt to perform both perception and multi-step planning latently, within a single forward pass. This coupled formulation, however, is bottlenecked by the LLM's limited capacity to discover and execute multi-step strategies in its latent representations. To address this bottleneck, we propose Hierarchical Programmatic Probing (HPP), a framework that decouples semantic perception from higher-order temporal reasoning by reformulating long video understanding as iterative, programmatic exploration of a hierarchically segmented video. Specifically, a coding-capable LLM plans and executes a multi-step strategy in an interactive coding environment, probing the video for information and invoking a VLM for localized perception on demand. To make probing tractable over long videos, we introduce three components: information-density-aware hierarchical segmentation, late-interaction semantic retrieval, and structured probing functions for coarse-to-fine temporal localization. We validate HPP on LongVideoBench, which requires both fine-grained perception and long-range relational reasoning, and show that decoupling the two via iterative programmatic probing yields substantial gains. Further results on EgoSchema, VideoMME, and MLVU demonstrate the effectiveness of our approach across diverse long-video benchmarks.

Problem

• long video understanding
• perception-reasoning decoupling
• multi-step reasoning
• spatio-temporal dynamics
• vision-language models

Innovation

• Hierarchical Programmatic Probing
• Decoupled Perception and Reasoning
• Long Video Understanding
• Interactive Coding Environment
• Information-Density-Aware Segmentation