Gaussian Process Prior Variational Autoencoder for Endoscopic Videos

2026-06-18

AI Summary

This work addresses the challenge of transient artifacts—such as specular reflections, motion blur, and missing frames—that commonly degrade endoscopic videos and hinder clinical interpretation and downstream tasks. To tackle this, the authors propose the Gaussian Process Prior Variational Autoencoder (GPVAE), which, for the first time, incorporates a temporal Gaussian process prior into an endoscopic video reconstruction framework, replacing the conventional assumption of independent latent variables. This enables coherent temporal modeling and uncertainty-aware inference. The model integrates an EndoVAE convolutional backbone with a GastroNet-5M pre-trained Vision Transformer encoder, employs scalable Gaussian process approximations via HPA/SPA, and leverages DUCKNet for mask-guided handling of specular reflections. Evaluated on C3VDv2, GPVAE reduces image reconstruction RMSE by 21.9% on average (up to 26.1%) and decreases downstream trajectory RMSE by 12.7%, while providing reliable frame-level confidence estimates.

Abstract

Endoscopic video analysis is essential for gastrointestinal diagnosis and computer-assisted interventions, but video sequences are routinely degraded by specular reflections, motion artifacts, and missing frames. These transient corruptions can distract clinicians, reduce image interpretability, and disrupt downstream tasks such as 3D reconstruction and navigation. Effective restoration therefore requires methods that exploit temporal continuity rather than treating frames in isolation. We introduce a Gaussian Process Prior Variational Autoencoder (GPVAE) framework for endoscopic video restoration that replaces the standard factorized latent prior with a temporal Gaussian process prior, enabling interpolation of missing frames with uncertainty-aware reconstruction. The framework combines endoscopy-specific encoders, including a convolutional EndoVAE backbone and pretrained Vision Transformer encoders from GastroNet-5M, with two scalable GP approximations: Hierarchical Prior Approximation (HPA) and Sparse Precision Approximation (SPA). Specular reflections are handled using a DUCKNet-based masking pipeline that excludes corrupted pixels from the reconstruction objective. On the C3VDv2 colonoscopy dataset, the best GPVAE variants reduced image reconstruction RMSE by 21.9\% on average, and by up to 26.1\%, relative to matched VAE baselines. Downstream trajectory RMSE was reduced by 12.7\% on average across classical visual odometry and a pretrained PoseNet, at an average increase of 27.3\% in training time per epoch. Finally, the GP posterior provides per-frame uncertainty estimates that reflect temporal support and offer a confidence signal for restored frames.

Problem

• endoscopic video restoration
• specular reflections
• missing frames
• motion artifacts
• temporal continuity

Innovation

• Gaussian Process Prior
• Variational Autoencoder
• Temporal Continuity
• Uncertainty-aware Reconstruction
• Endoscopic Video Restoration