P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecture

2026-06-22

AI Summary

Existing video foundation models struggle to effectively model long-horizon procedural videos due to the high computational complexity of self-attention mechanisms and limited ability to distinguish visually similar yet semantically distinct actions. To address these limitations, this work proposes a backbone-agnostic, dense frame-aligned action modeling approach that enables efficient long-video understanding by predicting masked pooled latent vectors. Built upon a predictive joint embedding architecture (P-JEPA) and leveraging features from VJEPA2.1, TSM, and I3D, the model is trained on datasets such as EgoExo4D. It achieves state-of-the-art performance on fine-grained action classification tasks while using an order of magnitude fewer parameters than large language model–based approaches. Furthermore, the method supports real-time streaming inference and temporally precise segmentation.

Abstract

The increasing maturity of embodied AI platforms has driven a growing interest in procedural video representation learning to support intelligent assistance systems for complex, multi-step tasks. Leveraging large-scale latent predictive training, video foundation models capture video dynamics, enabling downstream tasks such as activity understanding, spatiotemporal localization, and predictive control. However, procedural videos include actions with long-range dependencies that these models do not support, due to the quadratic complexity of self-attention. Distinct actions, for example, may be visually similar despite appearing at different points in the procedure, such as turning the stove on versus off. Here, we propose a backbone-agnostic approach that learns long-duration video representations by reducing the problem to a dense, frame-aligned action space and predicting pooled masked latent vectors. This approach allows our Procedural Joint Embedding Predictive Architecture (P-JEPA) to ingest videos over 30 minutes long, enabling effective long-form understanding of procedural steps. We evaluate P-JEPA using features extracted with VJEPA2.1, TSM, and I3D over the EgoExo4D, EgoProceL, and Assembly101 datasets, finding that it consistently improves linear separability, streaming inference, and temporal action segmentation performance, achieving state-of-the-art results on EgoExo4D fine-grained action classification while using an order of magnitude fewer parameters than LLM-based methods and running in real time.

Problem

• procedural video
• long-range dependencies
• video representation learning
• action understanding
• temporal action segmentation

Innovation

• procedural video understanding
• long-range dependency
• masked latent prediction
• frame-aligned action space
• backbone-agnostic architecture