AVA-Encoder: Towards Agent-Native Video Representation Learning

2026-08-12

AI Summary

This work addresses the challenge that existing creative agents struggle to effectively learn from high-quality films, primarily due to the absence of structured video representations that are both content-faithful and amenable to agent-based reasoning and manipulation. To bridge this gap, the authors propose the AVA-Encoder framework, which introduces the first native video knowledge graph representation tailored for agent operation. This framework enables bidirectional conversion between video and knowledge graph through a self-encoding mechanism and incorporates a text-gradient optimization strategy guided by natural language update directions. The contributions include the first cinematic-scale knowledge graph dataset, a reconstruction benchmark, and a technical pipeline integrating multimodal asset linking, typed edge relations, and data-agnostic encoding. Experiments demonstrate that the method outperforms the strongest baseline by 20.7 percentage points in video reconstruction and achieves superior performance over hand-tuned strategies using 74.3% fewer prompts under a policy-only setting.

Abstract

Creative agents still lack an effective way to learn from high-quality human films, limiting their ability to produce cinematic-grade videos. A key challenge is the absence of a structured video representation that is both faithful to film content and directly usable for agentic reasoning and manipulation. To address the challenge, we propose the Agentic Video Auto-Encoder (AVA-Encoder), a framework for learning agent-native video representations via agentic auto-encoding. AVA-Encoder transforms a video into a knowledge graph (KG) representation and then reconstructs it back into video. Its hierarchy and state nodes store structured text, while a linked asset layer holds generated images, audio, and video. Typed edges preserve the relations between these text descriptions and assets in a form that agents can easily understand, query, and edit. The video reconstruction differences drive a textual-gradient optimization framework, which expresses evaluation feedback as natural-language update directions for Data-Independent Encoding Policy Pseudo-Training in the outer loop and optional Data-Dependent KG Representation Refinement in the test-time inner loop. Extensive experiments show that AVA-Encoder improves by 20.7 percentage points over the strongest external baseline. In the controlled policy-only setting, its pseudo-trained shot-level Agentic Video Encoder policy also outperforms a carefully human-tuned policy while using 74.3% fewer system-prompt tokens. We release the complete AVA-Encoder framework, a reliable agentic video reconstruction benchmark, and the first dataset of high-quality film KG representations.

Problem

• agent-native video representation
• video representation learning
• agentic reasoning
• structured video representation
• cinematic-grade video generation

Innovation

• agent-native representation
• video auto-encoding
• knowledge graph
• textual-gradient optimization
• agentic video understanding