Emergence of Abstractions: Concept Encoding and Decoding Mechanism for In-Context Learning in Transformers

2024-12-16

arXiv.org

AI Summary

This study investigates how autoregressive Transformers achieve implicit task adaptation via in-context learning (ICL), focusing on the formation mechanism of task vectors during pretraining and their representational quality as predictors of ICL performance. Method: We propose the “concept encoding–decoding coupling” hypothesis and empirically validate it through representational dynamics analysis on synthetic ICL tasks across Gemma-2 and Llama-3.1 models (2B–70B parameters). Contribution/Results: We demonstrate that abstract task representations and conditional decoding algorithms co-emerge during pretraining; their coupling strength causally governs ICL accuracy, and a quantified coupling metric predicts task performance with high fidelity. This work breaks from the black-box paradigm, establishing the first representation-level ICL framework with causal interpretability and cross-model generality—unifying mechanistic understanding of implicit task adaptation across model scales.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Humans distill complex experiences into fundamental abstractions that enable rapid learning and adaptation. Similarly, autoregressive transformers exhibit adaptive learning through in-context learning (ICL), which begs the question of how. In this paper, we propose concept encoding-decoding mechanism to explain ICL by studying how transformers form and use internal abstractions in their representations. On synthetic ICL tasks, we analyze the training dynamics of a small transformer and report the coupled emergence of concept encoding and decoding. As the model learns to encode different latent concepts (e.g., ``Finding the first noun in a sentence.") into distinct, separable representations, it concureently builds conditional decoding algorithms and improve its ICL performance. We validate the existence of this mechanism across pretrained models of varying scales (Gemma-2 2B/9B/27B, Llama-3.1 8B/70B). Further, through mechanistic interventions and controlled finetuning, we demonstrate that the quality of concept encoding is causally related and predictive of ICL performance. Our empirical insights shed light into better understanding the success and failure modes of large language models via their representations.

Problem

• How transformers form task vectors during pretraining
• How task encoding quality predicts ICL performance
• Impact of layer finetuning on task encoding and performance

Innovation

• Encodes tasks into distinct separable representations
• Builds conditional decoding algorithms for tasks
• Finetunes earlier layers to improve task encoding