Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training

2026-05-10

AI Summary

This work addresses catastrophic forgetting in large language models during continual post-training and the absence of effective criteria to assess the utility of updates. From a task geometry perspective, the study reveals that forgetting arises from a misalignment between the geometry induced by new tasks and the model’s current representational geometry, introducing geometric conflict as an interpretable and controllable signal for forgetting. Building on this insight, the authors propose a replay-free, geometry-aware fusion mechanism that constructs a shared metric space via Gaussian Wasserstein barycenters and integrates parameter updates through a geometric conflict gating strategy, entirely without relying on stored data. Experiments on Qwen3 models (0.6B–14B) demonstrate that the method significantly outperforms existing data-free baselines in both domain- and capability-based continual learning settings, simultaneously improving knowledge retention and final performance.

Abstract

Continual post-training aims to extend large language models (LLMs) with new knowledge, skills, and behaviors, yet it remains unclear when sequential updates enable capability transfer and when they cause catastrophic forgetting. Existing methods mitigate forgetting through sequential fine-tuning, replay, regularization, or model merging, but offer limited criteria for determining when incorporating new updates is beneficial or harmful. In this work, we study LLM continual post-training through three questions: What drives forgetting? When do sequentially acquired capabilities transfer or interfere? How can compatibility be used to control update integration? We address these questions through task geometry: we represent each post-training task by its parameter update and study the covariance geometry induced by the update. Our central finding is that: forgetting can be considered as a state-relative update-integration failure, it arises when the covariance geometries induced by tasks misalign with the geometry of the evolving model state. Sequential updates transfer when they remain compatible with the model state shaped by previous updates, and interfere when state-relative geometry conflict becomes high. Motivated by this finding, we propose Geometry-Conflict Wasserstein Merging (GCWM), a data-free update-integration method that constructs a shared Wasserstein metric via Gaussian Wasserstein barycenters and uses geometry conflict to gate geometry-aware correction. Across Qwen3 0.6B--14B on domain-continual and capability-continual settings, GCWM consistently outperforms data-free baselines, improving retention and final performance without replay data. These results identify geometry conflict as both an explanatory signal for forgetting and a practical control signal for LLM continual post-training.

Problem

• catastrophic forgetting
• continual post-training
• capability transfer
• geometry conflict
• large language models

Innovation

• geometry conflict
• continual post-training
• Wasserstein merging
• catastrophic forgetting
• task geometry