Mechanistic understanding and validation of large AI models with SemanticLens

2025-01-09

AI Summary

To address the opacity and verification challenges inherent in large models—particularly neural networks—this paper proposes SemanticLens, a general, fully automated, component-level semantic interpretation framework. SemanticLens maps hidden-layer neurons into a multimodal semantic space grounded in foundation models (e.g., CLIP), enabling concept localization, functional attribution, and regulatory compliance auditing via zero-shot semantic retrieval, neuron–concept alignment modeling, and structured representation comparison. Its key innovations include the first text-driven concept neuron search, automatic alignment with clinical guidelines (e.g., the ABCDE melanoma diagnostic criteria), and bias auditing. Evaluated on skin cancer classification, SemanticLens successfully identifies spurious correlations, debugs black-box models, and quantifies knowledge distribution across neurons. The framework substantially enhances model interpretability, decision trustworthiness, and clinical verifiability—bridging critical gaps between AI behavior and domain-specific medical reasoning.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Unlike human-engineered systems such as aeroplanes, where each component's role and dependencies are well understood, the inner workings of AI models remain largely opaque, hindering verifiability and undermining trust. This paper introduces SemanticLens, a universal explanation method for neural networks that maps hidden knowledge encoded by components (e.g., individual neurons) into the semantically structured, multimodal space of a foundation model such as CLIP. In this space, unique operations become possible, including (i) textual search to identify neurons encoding specific concepts, (ii) systematic analysis and comparison of model representations, (iii) automated labelling of neurons and explanation of their functional roles, and (iv) audits to validate decision-making against requirements. Fully scalable and operating without human input, SemanticLens is shown to be effective for debugging and validation, summarizing model knowledge, aligning reasoning with expectations (e.g., adherence to the ABCDE-rule in melanoma classification), and detecting components tied to spurious correlations and their associated training data. By enabling component-level understanding and validation, the proposed approach helps bridge the"trust gap"between AI models and traditional engineered systems. We provide code for SemanticLens on https://github.com/jim-berend/semanticlens and a demo on https://semanticlens.hhi-research-insights.eu.

Problem

• Interpreting AI Models
• Neural Networks
• Enhancing Trust and Verifiability

Innovation

• SemanticLens
• neural networks explanation
• multimodal semantic space