DINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classification

2026-04-08

AI Summary

Although vision foundation models such as DINOv2 exhibit strong representational capabilities, their high-dimensional features lack interpretability, hindering transparent image classification. To address this, this work proposes DINO-QPM—a lightweight, interpretable adapter that operates with a frozen DINOv2 backbone. By replacing the CLS token with average pooling to preserve spatial information and introducing a Quadratic Programming Module (QPM), the method disentangles entangled features into class-agnostic, human-understandable contrastive representations. DINO-QPM is the first to adapt QPM to a frozen vision foundation model, incorporating a sparsity loss to suppress background noise and enhance explanation focus. Experiments demonstrate that, without fine-tuning the backbone, DINO-QPM surpasses the linear probe of DINOv2 in classification accuracy and significantly outperforms existing methods across multiple interpretability metrics, including a newly proposed Plausibility metric.

Abstract

Although visual foundation models like DINOv2 provide state-of-the-art performance as feature extractors, their complex, high-dimensional representations create substantial hurdles for interpretability. This work proposes DINO-QPM, which converts these powerful but entangled features into contrastive, class-independent representations that are interpretable by humans. DINO-QPM is a lightweight interpretability adapter that pursues globally interpretable image classification, adapting the Quadratic Programming Enhanced Model (QPM) to operate on strictly frozen DINO backbones. While classification with visual foundation models typically relies on the \texttt{CLS} token, we deliberately diverge from this standard. By leveraging average-pooling, we directly connect the patch embeddings to the model's features and therefore enable spatial localisation of DINO-QPM's globally interpretable features within the input space. Furthermore, we apply a sparsity loss to minimise spatial scatter and background noise, ensuring that explanations are grounded in relevant object parts. With DINO-QPM we make the level of interpretability of QPM available as an adapter while exceeding the accuracy of DINOv2 linear probe. Evaluated through an introduced Plausibility metric and other interpretability metrics, extensive experiments demonstrate that DINO-QPM is superior to other applicable methods for frozen visual foundation models in both classification accuracy and explanation quality.

Problem

• interpretability
• visual foundation models
• image classification
• feature disentanglement
• explainable AI

Innovation

• interpretable AI
• visual foundation models
• feature disentanglement
• sparsity loss
• frozen backbone adaptation