Towards an Explainable Comparison and Alignment of Feature Embeddings

2025-06-06

AI Summary

This paper addresses the lack of interpretable comparison and alignment mechanisms among multi-feature embedding models. We propose the Spectral Pairwise Embedding Comparison (SPEC) framework, which performs spectral analysis on differential kernel matrices to localize regions of inconsistent sample clustering across embedding spaces—enabling interpretable diagnosis of embedding discrepancies. SPEC further introduces an end-to-end convex optimization procedure for embedding alignment that preserves semantic consistency while achieving linear time complexity (O(n)). To our knowledge, SPEC is the first framework to establish a feature-decomposition–based paradigm for interpretable embedding comparison, uniquely balancing theoretical rigor, computational scalability, and interpretability. Extensive experiments on large-scale benchmarks—including ImageNet and MS-COCO—demonstrate its effectiveness. The implementation is publicly available.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Machine Learning: Semi-Supervised Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While several feature embedding models have been developed in the literature, comparisons of these embeddings have largely focused on their numerical performance in classification-related downstream applications. However, an interpretable comparison of different embeddings requires identifying and analyzing mismatches between sample groups clustered within the embedding spaces. In this work, we propose the emph{Spectral Pairwise Embedding Comparison (SPEC)} framework to compare embeddings and identify their differences in clustering a reference dataset. Our approach examines the kernel matrices derived from two embeddings and leverages the eigendecomposition of the difference kernel matrix to detect sample clusters that are captured differently by the two embeddings. We present a scalable implementation of this kernel-based approach, with computational complexity that grows linearly with the sample size. Furthermore, we introduce an optimization problem using this framework to align two embeddings, ensuring that clusters identified in one embedding are also captured in the other model. We provide numerical results demonstrating the SPEC's application to compare and align embeddings on large-scale datasets such as ImageNet and MS-COCO. The code is available at [https://github.com/mjalali/embedding-comparison](github.com/mjalali/embedding-comparison).

Problem

• Compare feature embeddings interpretably beyond numerical performance
• Identify mismatches in sample clusters between different embeddings
• Align embeddings to ensure consistent clustering across models

Innovation

• SPEC framework compares embeddings via kernel matrices
• Scalable kernel-based approach with linear complexity
• Optimization aligns embeddings to match clusters