Vega-Video: Integrating Video into the Grammar of Graphics

2026-04-27

AI Summary

This work addresses the challenges of integrating video with traditional data in visualization—namely, substantial paradigmatic differences, high performance bottlenecks, and difficulties in achieving seamless integration. The authors propose a unified framework grounded in Vega’s declarative grammar, formalizing video visualization into three core operations: synchronization, annotation, and transformation. They introduce a novel split-signal architecture that conceals video update latency while preserving declarative semantics. Furthermore, the framework enables encoding-aware optimization by detecting continuous drag interactions at compile time and leverages existing VOD streaming protocols to support efficient video transformations. Evaluated on multi-hour videos, the approach achieves sub-200-millisecond real-time responsiveness and demonstrates a fourfold improvement in interactive performance, marking the first effective and seamless integration of video and conventional data in visual analytics.

Abstract

Video data is increasingly used alongside conventional data for interactive data exploration, necessitating interfaces for exploring and presenting mixed-modality data. However, integrating video into visualizations remains difficult due to its distinct paradigms and inherent performance challenges. We identify three classes of video data visualization - synchronization, annotation, and transformation - and integrate them into the Vega declarative grammar. We show that these abstractions enable high-performance implementation. To reconcile Vega's instantaneous dataflow with video player state, we introduce a split-signal architecture that preserves declarative semantics while masking video update delays. We detect continuous scrubbing interactions at compile time to apply encoding-aware optimizations that improve responsiveness by up to 4x. We also repurpose VOD protocols to transform videos in real time, delivering sub-200ms updates even on multi-hour-long compilations. These contributions enable seamless integration of conventional and video data visualization.

Problem

• video visualization
• grammar of graphics
• mixed-modality data
• interactive exploration
• declarative visualization

Innovation

• video visualization
• declarative grammar
• split-signal architecture
• encoding-aware optimization
• real-time video transformation