Robust Foreground-Background Separation for Severely-Degraded Videos Using Convolutional Sparse Representation Modeling

2025-06-21

AI Summary

Existing foreground-background separation methods struggle with severely degraded videos characterized by low frame rates and concurrent multiple noise types, failing to simultaneously capture data-specific characteristics and generalizable features while lacking explicit noise modeling. Method: We propose a convolutional sparse representation-based foreground modeling framework that innovatively integrates data-driven specificity learning with prior-guided generic feature extraction, augmented by a differentiable multi-noise explicit modeling function. Within a constrained multi-convex optimization framework, we design an alternating direction algorithm that jointly achieves precise foreground extraction and robust noise suppression. Results: Experiments on infrared and microscopic video datasets demonstrate significant improvements over state-of-the-art methods, achieving new SOTA performance in both separation accuracy and noise robustness.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Non-convex Optimization

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

This paper proposes a foreground-background separation (FBS) method with a novel foreground model based on convolutional sparse representation (CSR). In order to analyze the dynamic and static components of videos acquired under undesirable conditions, such as hardware, environmental, and power limitations, it is essential to establish an FBS method that can handle videos with low frame rates and various types of noise. Existing FBS methods have two limitations that prevent us from accurately separating foreground and background components from such degraded videos. First, they only capture either data-specific or general features of the components. Second, they do not include explicit models for various types of noise to remove them in the FBS process. To this end, we propose a robust FBS method with a CSR-based foreground model. This model can adaptively capture specific spatial structures scattered in imaging data. Then, we formulate FBS as a constrained multiconvex optimization problem that incorporates CSR, functions that capture general features, and explicit noise characterization functions for multiple types of noise. Thanks to these functions, our method captures both data-specific and general features to accurately separate the components from various types of noise even under low frame rates. To obtain a solution of the optimization problem, we develop an algorithm that alternately solves its two convex subproblems by newly established algorithms. Experiments demonstrate the superiority of our method over existing methods using two types of degraded videos: infrared and microscope videos.

Problem

• Separate foreground-background in severely-degraded videos
• Handle low frame rates and various noise types
• Improve accuracy with convolutional sparse representation

Innovation

• CSR-based foreground model for spatial structures
• Multiconvex optimization with noise characterization
• Alternate algorithm for solving subproblems