🤖 AI Summary
This work investigates the mechanistic role of induction heads in enabling in-context learning (ICL) in large language models (LLMs). To establish causality—not mere correlation—we conduct targeted ablation experiments and attention masking across two distinct architectures (Llama-3-8B and InternLM2-20B). Our analysis demonstrates that induction heads are strictly necessary for ICL: their removal degrades performance on abstract reasoning tasks by 32%, reducing accuracy to near-chance levels, and collapses NLP task performance to zero-shot baselines. This is the first causal validation of induction heads’ functional necessity in ICL. By elevating their status from statistically associated features to indispensable computational primitives, our findings identify induction heads as a core neuro-symbolic module underpinning implicit reasoning in LLMs. The results provide interpretable, experimentally verifiable evidence for how LLMs perform pattern matching and generalization without explicit parameter updates—advancing the mechanistic understanding of in-context inference.
📝 Abstract
Large language models (LLMs) have shown a remarkable ability to learn and perform complex tasks through in-context learning (ICL). However, a comprehensive understanding of its internal mechanisms is still lacking. This paper explores the role of induction heads in a few-shot ICL setting. We analyse two state-of-the-art models, Llama-3-8B and InternLM2-20B on abstract pattern recognition and NLP tasks. Our results show that even a minimal ablation of induction heads leads to ICL performance decreases of up to ~32% for abstract pattern recognition tasks, bringing the performance close to random. For NLP tasks, this ablation substantially decreases the model's ability to benefit from examples, bringing few-shot ICL performance close to that of zero-shot prompts. We further use attention knockout to disable specific induction patterns, and present fine-grained evidence for the role that the induction mechanism plays in ICL.