Explainable Mapper: Charting LLM Embedding Spaces Using Perturbation-Based Explanation and Verification Agents

2025-07-24

AI Summary

To address the low efficiency of manual analysis and poor semantic interpretability of large language model (LLM) embedding space topology, this paper proposes a customizable dual-agent mapping graph analysis framework. One agent employs perturbation-driven LLM inference to enable semi-automatic semantic labeling and robustness verification of embedding features; the other integrates topological data analysis (TDA) with an interactive visualization system for exploratory analysis of topological elements—including nodes, paths, and clusters. The framework establishes a taxonomy of explorable elements within the mapping graph and constructs semantics-preserving topological representations of the embedding space using the Mapper algorithm. Experiments on multi-layer BERT embeddings successfully reproduce known linguistic properties and uncover novel topological-neighborhood semantic patterns. Results demonstrate significant improvements in interpretability, automation level, and scalability compared to prior approaches.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) produce high-dimensional embeddings that capture rich semantic and syntactic relationships between words, sentences, and concepts. Investigating the topological structures of LLM embedding spaces via mapper graphs enables us to understand their underlying structures. Specifically, a mapper graph summarizes the topological structure of the embedding space, where each node represents a topological neighborhood (containing a cluster of embeddings), and an edge connects two nodes if their corresponding neighborhoods overlap. However, manually exploring these embedding spaces to uncover encoded linguistic properties requires considerable human effort. To address this challenge, we introduce a framework for semi-automatic annotation of these embedding properties. To organize the exploration process, we first define a taxonomy of explorable elements within a mapper graph such as nodes, edges, paths, components, and trajectories. The annotation of these elements is executed through two types of customizable LLM-based agents that employ perturbation techniques for scalable and automated analysis. These agents help to explore and explain the characteristics of mapper elements and verify the robustness of the generated explanations. We instantiate the framework within a visual analytics workspace and demonstrate its effectiveness through case studies. In particular, we replicate findings from prior research on BERT's embedding properties across various layers of its architecture and provide further observations into the linguistic properties of topological neighborhoods.

Problem

• Understanding topological structures of LLM embedding spaces
• Reducing human effort in exploring linguistic properties
• Automating annotation of embedding properties via LLM agents

Innovation

• Perturbation-based LLM agents for analysis
• Semi-automatic annotation of embedding properties
• Visual analytics workspace for exploration