DocVCE: Diffusion-based Visual Counterfactual Explanations for Document Image Classification

2025-08-06

AI Summary

Poor interpretability of black-box document image classifiers and the limited semantic clarity of existing feature importance maps motivate this work. We introduce, for the first time, generative counterfactual explanations to document image classification, proposing a classifier-guided visual counterfactual generation method based on Latent Diffusion Models (LDMs). Our approach performs hierarchical block-wise fine-grained search in the latent space to optimize local patches, yielding high-fidelity counterfactuals that closely adhere to the natural data distribution. The method is agnostic to backbone architectures—supporting ResNet, ConvNeXt, and DiT—and achieves significant improvements in counterfactual validity, proximity, and realism on RVL-CDIP, Tobacco3482, and DocLayNet. By producing semantically meaningful, human-interpretable, and empirically verifiable counterfactuals, our framework enhances transparency and trustworthiness in black-box document AI systems.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

As black-box AI-driven decision-making systems become increasingly widespread in modern document processing workflows, improving their transparency and reliability has become critical, especially in high-stakes applications where biases or spurious correlations in decision-making could lead to serious consequences. One vital component often found in such document processing workflows is document image classification, which, despite its widespread use, remains difficult to explain. While some recent works have attempted to explain the decisions of document image classification models through feature-importance maps, these maps are often difficult to interpret and fail to provide insights into the global features learned by the model. In this paper, we aim to bridge this research gap by introducing generative document counterfactuals that provide meaningful insights into the model's decision-making through actionable explanations. In particular, we propose DocVCE, a novel approach that leverages latent diffusion models in combination with classifier guidance to first generate plausible in-distribution visual counterfactual explanations, and then performs hierarchical patch-wise refinement to search for a refined counterfactual that is closest to the target factual image. We demonstrate the effectiveness of our approach through a rigorous qualitative and quantitative assessment on 3 different document classification datasets -- RVL-CDIP, Tobacco3482, and DocLayNet -- and 3 different models -- ResNet, ConvNeXt, and DiT -- using well-established evaluation criteria such as validity, closeness, and realism. To the best of the authors' knowledge, this is the first work to explore generative counterfactual explanations in document image analysis.

Problem

• Explain decisions of document image classification models
• Generate plausible visual counterfactual explanations for documents
• Improve transparency of black-box AI in document processing

Innovation

• Leverages latent diffusion models for explanations
• Uses classifier guidance for counterfactual generation
• Performs hierarchical patch-wise refinement