Back to the Feature: Explaining Video Classifiers with Video Counterfactual Explanations

2025-11-25

AI Summary

Existing image-level counterfactual explanation (CFE) methods fail to generate physically plausible, temporally coherent, and motion-smooth CFEs for video classifiers. To address this, we propose BTTF—a bidirectional temporal trajectory fine-tuning framework—that initializes latent noise conditioned on the first frame and performs two-stage latent-space optimization without auxiliary networks, enabling high-fidelity, temporally consistent video CFE generation. Leveraging a classifier-free diffusion model with progressive denoising step scheduling, BTTF efficiently searches for the minimal perturbation sequence that flips the predicted class. Evaluations on Shape-Moving, MEAD, and NTU RGB+D demonstrate that BTTF-generated CFEs achieve superior visual realism, cross-model classification transferability, and intuitive interpretability—significantly advancing explainability for video understanding models.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Transparent, Interpretable, Explainable ML

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Counterfactual explanations (CFEs) are minimal and semantically meaningful modifications of the input of a model that alter the model predictions. They highlight the decisive features the model relies on, providing contrastive interpretations for classifiers. State-of-the-art visual counterfactual explanation methods are designed to explain image classifiers. The generation of CFEs for video classifiers remains largely underexplored. For the counterfactual videos to be useful, they have to be physically plausible, temporally coherent, and exhibit smooth motion trajectories. Existing CFE image-based methods, designed to explain image classifiers, lack the capacity to generate temporally coherent, smooth and physically plausible video CFEs. To address this, we propose Back To The Feature (BTTF), an optimization framework that generates video CFEs. Our method introduces two novel features, 1) an optimization scheme to retrieve the initial latent noise conditioned by the first frame of the input video, 2) a two-stage optimization strategy to enable the search for counterfactual videos in the vicinity of the input video. Both optimization processes are guided solely by the target classifier, ensuring the explanation is faithful. To accelerate convergence, we also introduce a progressive optimization strategy that incrementally increases the number of denoising steps. Extensive experiments on video datasets such as Shape-Moving (motion classification), MEAD (emotion classification), and NTU RGB+D (action classification) show that our BTTF effectively generates valid, visually similar and realistic counterfactual videos that provide concrete insights into the classifier's decision-making mechanism.

Problem

• Generating counterfactual explanations for video classifiers remains largely underexplored
• Existing image-based methods cannot create temporally coherent and physically plausible video explanations
• Current approaches lack capacity to produce smooth motion trajectories in counterfactual videos

Innovation

• Generates video counterfactuals via optimization framework
• Uses two-stage optimization for temporal coherence
• Employs progressive denoising to accelerate convergence