TrajTok: Learning Trajectory Tokens enables better Video Understanding

2026-02-26

AI Summary

This work addresses the inefficiency and limited scalability of conventional video models that rely on fixed patch-based tokenization, which generates excessive redundant tokens. Unlike existing trajectory-based tokenization methods requiring complex, task-agnostic external segmentation and tracking pipelines, we propose TrajTok—the first end-to-end trainable video tokenizer. TrajTok implicitly clusters spatiotemporal pixels to produce semantically adaptive object trajectory tokens in a single forward pass, decoupling token count from video duration. Token generation is driven by semantic complexity, enabling dynamic granularity control and seamless integration into downstream architectures such as TrajViT2, TrajAdapter, and TrajVLM. Experiments show that TrajViT2, trained from scratch, achieves state-of-the-art accuracy on classification and retrieval tasks while matching the efficiency of advanced compression methods, with particularly pronounced advantages in long-form video understanding.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While recent trajectory-based tokenizers offer a promising solution by decoupling video duration from token count, they rely on complex external segmentation and tracking pipelines that are slow and task-agnostic. We propose TrajTok, an end-to-end video tokenizer module that is fully integrated and co-trained with video models for a downstream objective, dynamically adapting its token granularity to semantic complexity, independent of video duration. TrajTok contains a unified segmenter that performs implicit clustering over pixels in both space and time to directly produce object trajectories in a single forward pass. By prioritizing downstream adaptability over pixel-perfect segmentation fidelity, TrajTok is lightweight and efficient, yet empirically improves video understanding performance. With TrajTok, we implement a video CLIP model trained from scratch (TrajViT2). It achieves the best accuracy at scale across both classification and retrieval benchmarks, while maintaining efficiency comparable to the best token-merging methods. TrajTok also proves to be a versatile component beyond its role as a tokenizer. We show that it can be seamlessly integrated as either a probing head for pretrained visual features (TrajAdapter) or an alignment connector in vision-language models (TrajVLM) with especially strong performance in long-video reasoning.

Problem

• video tokenization
• trajectory-based representation
• redundant tokens
• scalability
• efficiency

Innovation

• TrajTok
• trajectory-based tokenization
• end-to-end video tokenizer
• implicit spatiotemporal clustering
• video understanding