What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data

2026-06-18

AI Summary

This study investigates the phenomenon of “emergent misalignment” in narrow fine-tuning, wherein alignment capabilities degrade broadly yet unevenly on unseen scenarios. Through a systematic analysis of the interplay among training dynamics, model priors, and data, the work identifies the root causes of this effect for the first time and demonstrates that pre-fine-tuning activation features can effectively predict post-fine-tuning alignment performance. Combining loss–alignment correlation analysis, learning rate scheduling experiments, activation subspace similarity measurements, and random vector controls, the authors reveal a high degree of overlap between the activation subspaces elicited by training and evaluation prompts, with the extent of this overlap strongly correlated with shifts in alignment performance.

Abstract

Emergent misalignment (EM) is a phenomenon in which models generalize with narrow fine-tuning, leading to broad (yet uneven) misalignment across evaluation questions. We study EM and its variability directly through the components of fine-tuning: training dynamics, model priors, and data. (1) We first explored how in-domain training loss relates to out-of-domain alignment scores across datasets and model families. Then, we tried to induce potential alternative local minima through different learning schedules for one narrow fine-tuning, but did not find strong runs with better broad alignment scores conditioned on similar or lower training loss. (2) We found that although the mean and standard deviations of the misaligned model scores are usually statistically different from those of the pre-trained model, there are some potential signals on overall positive correlation. The evaluation prompt-only activations from both the pre-trained and the original instruct models (prior to narrow fine-tuning) could predict fine-grained alignment scores after narrow fine-tuning. (3) Finally, we compared activation deltas before and after narrow fine-tuning and found moderate-to-high subspace overlap and similarity between the resulting activation shifts for training and evaluation prompts. Subspace overlaps between training and evaluation prompt activations correlate with their shifts' similarities when measuring with the last prompt-token activations. The train-evaluation data prompt overlap is controlled against overlap computed from random vectors and evaluation prompts activations.

Problem

• Emergent Misalignment
• Fine-tuning
• Model Alignment
• Training Dynamics
• Data Distribution

Innovation

• Emergent Misalignment
• Training Dynamics
• Model Priors
• Activation Subspace Overlap
• Fine-tuning Generalization