Linking forward-pass dynamics in Transformers and real-time human processing

2025-04-18

AI Summary

Traditional black-box AI evaluation relies solely on output probabilities, failing to model human real-time cognitive processing. Method: This study investigates whether layer-wise temporal dynamics during Transformer forward propagation can capture such processing, using layer-wise activation analysis and inter-layer time-series modeling, integrated with multimodal neurobehavioral data (eye-tracking, EEG, reading times) across five cross-domain regression and incremental prediction tasks in language and vision. Contribution/Results: We provide the first systematic evidence that pretrained Transformers (e.g., BERT, ViT) implicitly acquire cognition-relevant representational structures; their internal computational dynamics significantly improve prediction of human real-time processing metrics (p < 0.001), with robust cross-modal and cross-task consistency. These findings support treating AI models as interpretable “processing” models—offering a novel paradigm bridging artificial intelligence and cognitive science.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Modern AI models are increasingly being used as theoretical tools to study human cognition. One dominant approach is to evaluate whether human-derived measures (such as offline judgments or real-time processing) are predicted by a model's output: that is, the end-product of forward pass(es) through the network. At the same time, recent advances in mechanistic interpretability have begun to reveal the internal processes that give rise to model outputs, raising the question of whether models and humans might arrive at outputs using similar"processing strategies". Here, we investigate the link between real-time processing in humans and"layer-time"dynamics in Transformer models. Across five studies spanning domains and modalities, we test whether the dynamics of computation in a single forward pass of pre-trained Transformers predict signatures of processing in humans, above and beyond properties of the model's output probability distribution. We consistently find that layer-time dynamics provide additional predictive power on top of output measures. Our results suggest that Transformer processing and human processing may be facilitated or impeded by similar properties of an input stimulus, and this similarity has emerged through general-purpose objectives such as next-token prediction or image recognition. Our work suggests a new way of using AI models to study human cognition: not just as a black box mapping stimuli to responses, but potentially also as explicit processing models.

Problem

• Link Transformer layer-time dynamics to human real-time processing
• Compare AI and human processing strategies across domains
• Assess predictive power of layer dynamics beyond model outputs

Innovation

• Linking Transformer layer-time dynamics to human processing
• Using pre-trained Transformers to predict human cognition
• Comparing model and human processing strategies directly