Problem
Research questions and friction points this paper is trying to address.
large language models
concept probing
model interpretability
abstract concepts
embedding analysis
What are They Thinking? Delineation, Probing, and Tracking of Concepts in LLMs
🤖 AI Summary
Existing approaches struggle to monitor high-level abstract concepts within the internal representations of large language models in real time. This work proposes a lightweight linear probing framework that enables detection and cross-context tracking of abstract concepts at any network layer by constructing paired datasets with and without specific concepts. The method systematically integrates concept definition, probe training, and tracking into a unified pipeline, offering both generality and low computational overhead. Experimental results demonstrate high-accuracy identification of four distinct abstract concepts across three mainstream large language models, validating the framework’s effectiveness and strong generalization capability.
📝 Abstract
As the influence of LLMs expands, it is imperative to gain insight into their decisions. One way to do that is to develop probes that detect the presence or absence of a broad set of concepts within the embeddings computed in an LLM - which is what we might say a model is"thinking"about. Such probes should be low-cost and easily applicable to any LLM, so that monitoring for many concepts is possible during normal operation. In this paper, we take the first steps towards developing the capability of creating many such probes by defining and executing examples of the key tasks needed: first, the careful delineation of a concept through the creation of a dataset with the concept both present and then absent. Then, the training and testing of a set of linear probes to detect the concept on any layer of an LLM, including an exploration of the complexity of the probe needed. Finally, we show that such probes can track concepts across larger contexts. This is done with four separate concepts and three different LLMs. When this process is scaled to many more concepts, it will create the ability to easily monitor new models.
Innovation
Methods, ideas, or system contributions that make the work stand out.
concept probing
linear probes
large language models
concept tracking
model interpretability
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
