P3CA: Encoder-Agnostic Interpretation of Vision Foundation Model Embeddings via Spatial Probing

2026-08-10

AI Summary

This work addresses the limited ability of high-dimensional embeddings in medical image analysis to effectively probe local semantic structures. To overcome this, the authors propose Position-Prompted Principal Component Analysis (P3CA), which estimates local feature normalization and principal covariance directions from user-specified spatial regions and applies the resulting projection globally to visualize localized semantics. P3CA enables region-conditioned interpretability of embeddings from any vision foundation model without requiring encoder modifications, retraining, or task-specific labels. By integrating spatial prompting, local PCA, feature normalization, and an interactive 3D Slicer workflow (EmbedVision), P3CA uncovers local structures obscured by global PCA across diverse data modalities—including natural images, colorectal histopathology, and spatial transcriptomics—significantly enhancing pathological discriminability and enabling direct comparison between learned representations and empirically observed spatial patterns.

Abstract

Vision foundation models are increasingly used as reusable encoders in medical image computing, yet their high-dimensional spatial embeddings are difficult to inspect beyond downstream task performance or global dimensionality reduction. We propose position-prompted PCA (P3CA), an encoder-agnostic method for local probing of channel-rich spatial tensors. Given a user-selected spatial prompt, P3CA estimates the feature normalization and dominant covariance directions within that region, then applies the resulting projection to the full tensor to visualize where locally informative directions are expressed. This produces a region-conditioned representation lens without modifying the encoder, retraining, or requiring task-specific labels. We implement P3CA in EmbedVision, an interactive 3D Slicer-based workflow, and evaluate it across natural images, colorectal pathology foundation-model embeddings, and spatial transcriptomic tensors. Across these settings, prompted projections reveal local structure suppressed by global PCA, improve prompt-matched pathology discrimination from frozen three-dimensional projections, and support comparison between learned and measured spatial representations.

Problem

• vision foundation models
• spatial embeddings
• interpretability
• medical image computing
• high-dimensional representation

Innovation

• encoder-agnostic
• spatial probing
• position-prompted PCA
• foundation model interpretation
• local covariance analysis