Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations

2025-05-19

AI Summary

This work investigates the metacognitive capacity of large language models (LLMs)—specifically, their ability to monitor and regulate internal neural activations—to support AI safety evaluation. Method: We introduce a neurofeedback-based paradigm: using sentence–activation label pairs, we train LLMs to explicitly report and modulate activations along specified representational directions. We formally define and quantify the “metacognitive space” as the subspace of activations that models can reliably introspect. Contribution/Results: We find that LLMs effectively monitor only low-dimensional, high-variance, semantically interpretable activation subspaces. Under optimal conditions, activation reporting accuracy and targeted modulation success reach 82%. This study establishes the first framework for assessing neural activation controllability in LLMs, demonstrating that their metacognition is limited yet quantifiable. It provides a novel, operationally grounded dimension for AI safety auditing and introduces practical tools for evaluating internal model behavior.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) can sometimes report the strategies they actually use to solve tasks, but they can also fail to do so. This suggests some degree of metacognition -- the capacity to monitor one's own cognitive processes for subsequent reporting and self-control. Metacognitive abilities enhance AI capabilities but raise safety concerns, as models might obscure their internal processes to evade neural-activation-based oversight mechanisms designed to detect harmful behaviors. Given society's increased reliance on these models, it is critical that we understand the limits of their metacognitive abilities, particularly their ability to monitor their internal activations. To address this, we introduce a neuroscience-inspired neurofeedback paradigm designed to quantify the ability of LLMs to explicitly report and control their activation patterns. By presenting models with sentence-label pairs where labels correspond to sentence-elicited internal activations along specific directions in the neural representation space, we demonstrate that LLMs can learn to report and control these activations. The performance varies with several factors: the number of example pairs provided, the semantic interpretability of the target neural direction, and the variance explained by that direction. These results reveal a"metacognitive space"with dimensionality much lower than the model's neural space, suggesting LLMs can monitor only a subset of their neural mechanisms. Our findings provide empirical evidence quantifying metacognitive capabilities in LLMs, with significant implications for AI safety.

Problem

• Assessing LLMs' ability to monitor internal activations metacognitively
• Investigating safety risks from LLMs obscuring internal processes
• Quantifying metacognitive control via neuroscience-inspired neurofeedback paradigms

Innovation

• Neuroscience-inspired neurofeedback paradigm for LLMs
• Quantify LLMs' ability to report activations
• Control activation patterns via sentence-label pairs