MoRFI: Monotonic Sparse Autoencoder Feature Identification

2026-04-29

AI Summary

This work investigates the emergence of hallucinations in large language models during supervised fine-tuning when new knowledge is introduced, a phenomenon whose underlying mechanisms remain poorly understood. Through controlled fine-tuning experiments on closed-book question answering, the authors analyze residual stream activations and employ sparse autoencoders to identify latent directions causally linked to hallucinations. They propose the Monotonic Relationship Feature Identification (MoRFI) method, which systematically isolates hallucination-related features that exhibit monotonic responses with respect to the target attribute—a first in the field. Experiments on Llama 3.1 8B, Gemma 2 9B, and Mistral 7B v0.3 demonstrate that intervening on a single latent variable can effectively recover overwritten knowledge, confirming the generality and efficacy of the proposed approach.

Abstract

Large language models (LLMs) acquire most of their factual knowledge during the pre-training stage, through next token prediction. Subsequent stages of post-training often introduce new facts outwith the parametric knowledge, giving rise to hallucinations. While it has been demonstrated that supervised fine-tuning (SFT) on new knowledge may exacerbate the problem, the underlying mechanisms are still poorly understood. We conduct a controlled fine-tuning experiment, focusing on closed-book QA, and find latent directions that causally contribute to hallucinations. Specifically, we fine-tune Llama 3.1 8B, Gemma 2 9B and Mistral 7B v03 on seven distinct single QA datasets, controlling for the percentage of new knowledge and number of training epochs. By measuring performance on the test set, we validate that incrementally introducing new knowledge increases hallucinations, with the effect being more pronounced with prolonged training. We leverage pre-trained sparse autoencoders (SAEs) to analyze residual stream activations across various checkpoints for each model and propose Monotonic Relationship Feature Identification (MoRFI) for capturing causally relevant latents. MoRFI filters SAE features that respond monotonically to controlled fine-tuning data mixtures of a target property. Our findings show that exposure to unknown facts disrupts the model's ability to retrieve stored knowledge along a set of directions in the residual stream. Our pipeline reliably discovers them across distinct models, recovering knowledge through single-latent interventions.

Problem

• hallucination
• supervised fine-tuning
• residual stream
• latent directions
• factual knowledge

Innovation

• Monotonic Relationship Feature Identification
• sparse autoencoder
• hallucination
• residual stream
• controlled fine-tuning