Greedy Coordinate Diffusion: Effective and Semantically Coherent Adversarial Attacks via Diffusion Guidance

2026-06-13

AI Summary

This work addresses the degradation of safety mechanisms in aligned language models when fine-tuned on harmless tasks, a phenomenon lacking theoretical explanation in prior research. The paper introduces the Alignment Instability Condition (AIC), which formally characterizes three key geometric properties responsible for alignment collapse, and reveals that alignment degradation follows a quartic onset law along gradient flow. Through geometric analysis in parameter space, gradient flow dynamics, and estimation of the Fisher information matrix, the study theoretically demonstrates that static first-order safeguards can fail under gradient descent. It further shows that first-order gradient orthogonal updates merely create an illusion of safety, while second-order curvature effects are the primary driver of degradation. Experiments confirm that the Fisher information matrix serves as an effective proxy for quantifying safety degradation.

Abstract

Fine-tuning aligned language models on benign tasks (e.g. math tutoring) systematically breaks safety guardrails, even when training data contains no harmful content. While mechanistic approaches have shed light on where alignment resides in model weights, they do not by provide a general formal framework for deriving guarantees about when fine-tuning degrades it -- leaving the field without principled tools for predicting or preventing alignment collapse. We develop a local geometric framework through geometric analysis of parameter-space trajectories and apply it to understand the fragility of alignment in fine-tuning. While first-order analysis suggests orthogonal updates are safe, we prove this is illusory: the curvature of the fine-tuning loss induces second-order acceleration that can induce second-order drift into alignment-sensitive regions. We formalize a construct of our framework as the Alignment Instability Condition (AIC), three geometric properties that, when present, are sufficient to guarantee degradation. Our main result proves quartic onset of alignment degradation along gradient-flow trajectories, determined by how sharply alignment depends on specific parameters and how strongly tasks couple to these parameters. These findings yield formal sufficient conditions under which static first-order protection can fail under gradient descent. We further empirically validate the framework's foundations, showing that the Fisher Information Matrix provides a proxy for the degree of safety degradation across diverse fine-tuning.

Problem

• alignment collapse
• fine-tuning
• safety degradation
• language models
• adversarial robustness

Innovation

• alignment instability
• geometric analysis
• fine-tuning safety
• Fisher Information Matrix
• parameter-space trajectory