Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

2026-07-09

AI Summary

This work addresses the persistent gap between knowledge acquisition and effective utilization in large language models during fine-tuning—where models rapidly memorize new information yet often fail to deploy it in downstream reasoning tasks. The authors propose the “knowledge circuit misalignment” hypothesis, positing that newly acquired knowledge is not properly routed to computationally effective network layers. To test and mitigate this issue, they introduce a novel self-patching intervention that integrates cross-domain fine-tuning with dynamic internal representation analysis to precisely identify activation sites critical for generalization. Experimental results validate the hypothesis and demonstrate that a heuristic strategy derived from these insights recovers 58%–75% of the theoretical performance upper bound in cases of generalization failure.

Abstract

Fine-tuning LLMs to inject new knowledge faces a critical challenge: LLMs can quickly memorize new facts, yet fail to use them for downstream reasoning tasks. We formalize this failure as the \textit{\textbf{Knowing--Using Gap}}, characterized by an accuracy gap and a temporal lag between memorization and generalization. To understand this phenomenon, we fine-tune LLMs with unseen knowledge and monitor the spatial permeation dynamics of the knowledge internally using a novel intervention technique called self-patching. Self-patching identifies activation locations where relocating representations substantially improves failed generalization cases. These results are consistent with a knowledge-circuit misalignment hypothesis: memorized representations can exist internally but may not be routed to computation-effective layers. To demonstrate the practicality of this diagnostic finding, we design a simple heuristic strategy which recovers 58--75\% of the oracle headroom in generalization failure. Experiments are done cross-domain for the robustness of this finding.

Problem

• Knowing--Using Gap
• knowledge generalization
• large language models
• fine-tuning
• memorization

Innovation

• self-patching
• Knowing–Using Gap
• knowledge-circuit misalignment
• activation routing
• fine-tuning generalization