PISCO: Precise Video Instance Insertion with Sparse Control

2026-02-09

AI Summary

This work addresses the challenge of achieving precise and controllable instance insertion in AI-generated videos, which requires simultaneous satisfaction of spatiotemporal localization, physical consistency, preservation of original dynamics, and minimal user effort. To this end, we propose PISCO, a video diffusion model guided by sparse keyframe control that supports flexible inputs—including single frames, start-end frame pairs, or arbitrary sparse keyframes—and automatically propagates object appearance, motion, and scene interactions across the video. PISCO innovatively integrates variable information guidance, distribution-preserving temporal masking, and geometry-aware conditional control to effectively mitigate distribution shifts in pretrained diffusion models under sparse supervision. Evaluated on our newly curated PISCO-Bench benchmark, PISCO substantially outperforms existing video editing and inpainting methods, with performance monotonically improving as the number of control signals increases.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The landscape of AI video generation is undergoing a pivotal shift: moving beyond general generation - which relies on exhaustive prompt-engineering and"cherry-picking"- towards fine-grained, controllable generation and high-fidelity post-processing. In professional AI-assisted filmmaking, it is crucial to perform precise, targeted modifications. A cornerstone of this transition is video instance insertion, which requires inserting a specific instance into existing footage while maintaining scene integrity. Unlike traditional video editing, this task demands several requirements: precise spatial-temporal placement, physically consistent scene interaction, and the faithful preservation of original dynamics - all achieved under minimal user effort. In this paper, we propose PISCO, a video diffusion model for precise video instance insertion with arbitrary sparse keyframe control. PISCO allows users to specify a single keyframe, start-and-end keyframes, or sparse keyframes at arbitrary timestamps, and automatically propagates object appearance, motion, and interaction. To address the severe distribution shift induced by sparse conditioning in pretrained video diffusion models, we introduce Variable-Information Guidance for robust conditioning and Distribution-Preserving Temporal Masking to stabilize temporal generation, together with geometry-aware conditioning for realistic scene adaptation. We further construct PISCO-Bench, a benchmark with verified instance annotations and paired clean background videos, and evaluate performance using both reference-based and reference-free perceptual metrics. Experiments demonstrate that PISCO consistently outperforms strong inpainting and video editing baselines under sparse control, and exhibits clear, monotonic performance improvements as additional control signals are provided. Project page: xiangbogaobarry.github.io/PISCO.

Problem

• video instance insertion
• sparse control
• controllable video generation
• scene integrity
• temporal consistency

Innovation

• video instance insertion
• sparse keyframe control
• variable-information guidance
• distribution-preserving temporal masking
• geometry-aware conditioning