Show Me What You Don't Know: Efficient Sampling from Invariant Sets for Model Validation

2026-03-23

AI Summary

This work addresses the challenge of efficiently verifying invariance properties of machine learning models under task-irrelevant data perturbations while preserving sensitivity to task-relevant features. The authors propose a general, training-free method that leverages pre-trained diffusion or flow-matching models as priors to guide nonlinear denoising trajectories via a fiber loss, enabling efficient sampling within invariance fibers—i.e., equivalence classes—of feature extractors. Evaluated across diverse datasets including ImageNet and CheXpert, and models such as ResNet, DINO, and BiomedCLIP, the approach successfully uncovers a range of invariance behaviors, including concerning cases like Qwen-2B misclassifying mirrored anatomical structures as normal. This provides a novel tool for assessing model reliability through systematic invariance analysis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The performance of machine learning models is determined by the quality of their learned features. They should be invariant under irrelevant data variation but sensitive to task-relevant details. To visualize whether this is the case, we propose a method to analyze feature extractors by sampling from their fibers -- equivalence classes defined by their invariances -- given an arbitrary representative. Unlike existing work where a dedicated generative model is trained for each feature detector, our algorithm is training-free and exploits a pretrained diffusion or flow-matching model as a prior. The fiber loss -- which penalizes mismatch in features -- guides the denoising process toward the desired equivalence class, via non-linear diffusion trajectory matching. This replaces days of training for invariance learning with a single guided generation procedure at comparable fidelity. Experiments on popular datasets (ImageNet, CheXpert) and model types (ResNet, DINO, BiomedClip) demonstrate that our framework can reveal invariances ranging from very desirable to concerning behaviour. For instance, we show how Qwen-2B places patients with situs inversus (heart on the right side) in the same fiber as typical anatomy.

Problem

• invariance
• model validation
• feature extractor
• fiber sampling
• machine learning

Innovation

• invariant sets
• fiber sampling
• training-free validation
• diffusion guidance
• feature invariance analysis