What Pixels Are Enough? SEAMS: Sufficiency Saliency via MSE-Preservation Soft-Masks

2026-07-10

AI Summary

This work addresses the challenge of identifying image regions sufficient for a model’s output to generate reliable and interpretable saliency maps. The authors propose SEAMS, a method that directly optimizes soft masks to faithfully preserve specific outputs—such as class probabilities or embeddings—of frozen vision models like ViT-S/16 and ConvNeXt, without requiring external data or architectural modifications. SEAMS introduces a learnable budget constraint and a three-way image synthesis strategy, enabling object-level, class-conditional, and token-level explanations within a unified framework. The resulting masks are compact, stable, and robust to initialization, achieving strong performance on insertion and deletion benchmarks. Notably, the approach reveals that different architectures can attain comparable output fidelity while relying on distinct sets of sufficient evidence.

Abstract

Saliency maps are most useful when they identify the image regions that are sufficient to preserve a model's behaviour. We introduce SEAMS, a sufficiency-based saliency method that directly optimises a soft mask using a preservation objective. Given a frozen differentiable model output, such as a class probability, CLS embedding, or token representation, SEAMS searches for a compact mask that preserves the selected output. The approach relies on a simple optimisation framework based on soft masks, a learnable budget, and a three-way image composite generated entirely from the query image. As a result, it requires no auxiliary distractor dataset, architecture-specific attribution mechanism, or differentiable top-k relaxation. Experiments with frozen ViT-S/16 and ConvNeXt models show that the same optimisation pipeline can generate object-level, class-conditioned, and token-level explanations by changing only the preserved target. The resulting masks are compact, interpretable, stable across random initialisations, and competitive on insertion and deletion benchmarks. Our results also indicate that different architectures often rely on different sufficient evidence while achieving similar preservation fidelity, highlighting the architecture-dependent nature of visual explanations.

Problem

• saliency
• sufficiency
• model explanation
• visual interpretation
• preservation

Innovation

• sufficiency-based saliency
• soft mask optimization
• model-agnostic explanation
• MSE-preservation
• visual interpretability