The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology

2026-07-01

AI Summary

Current interpretability research often relies on mechanistic models (MOs) constructed via post-hoc supervised fine-tuning, which may overestimate their interpretability and fail to reflect real-world scenarios. This work systematically constructs 54 MOs based on OLMo2-1B and Gemma-3-1B-it, spanning seven training paradigms—including supervised fine-tuning (SFT), direct preference optimization (DPO), activation predictors, and activation manipulation—and reveals for the first time that MO interpretability is significantly influenced by training objectives, behavioral types, model architectures, and data generation pipelines. The study introduces a more realistic integrated training paradigm and finds that MOs trained under such settings are generally less interpretable than those trained post-hoc, thereby challenging the validity of current MOs as proxies for interpretability.

Abstract

Model organisms (MOs) - language models trained to exhibit undesired or unnatural behaviours - are frequently used as testbeds for evaluating white-box interpretability techniques. Current MOs are typically constructed via post-hoc supervised fine-tuning (SFT) on behavioural transcripts or synthetic documents. Prior research has shown that interpretability methods can easily identify hidden behaviours in these MOs. However, recent work suggests that such post-hoc training methods may make interpretability unrealistically easy. We investigate this claim by constructing a suite of 54 $\verb|OLMo2-1B|$- and $\verb|gemma-3-1b-it|$-based MOs trained with seven different techniques, including standard post-hoc SFT, post-hoc DPO, and more realistic integration of MO data into the OLMo post-training DPO phase. We use these MO variants to benchmark activation oracles, activation steering, logit lens, and sparse autoencoders. Our findings show that (i) MO interpretability depends strongly on training objective, target behaviour, model architecture, and training data generation pipeline; (ii) substantial variance remains even after controlling for differences in the strength of target behaviour expression; and (iii) our more realistic $\textit{integrated training}$ often yields less interpretable MOs than standard post-hoc methods. Our results cast substantial doubt on the validity of current MOs as interpretability proxies.

Problem

• Model Organisms
• Interpretability
• Training Methodology
• White-box Interpretability
• Fine-tuning

Innovation

• Model Organisms
• Interpretability
• Integrated Training
• DPO
• Activation Steering