Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgetting

2025-08-07

AI Summary

This work addresses the limitation that pretrained model weights inherit knowledge capacity constrained by the original pretraining dataset. To overcome this, we propose KNOW—a novel prediction method that for the first time models knowledge forgetting during fine-tuning as a reversible dynamical system, enabling backward inference to recover “forgotten” knowledge from sequentially pruned fine-tuning trajectories. Methodologically, we construct a controlled-forgetting weight evolution dataset and employ meta-learning to train a hypermodel, KNOWN, which jointly models weight dynamics and performs enhanced knowledge extrapolation. Extensive experiments across diverse datasets and architectures demonstrate that the resulting augmented pretrained weights substantially outperform both naive fine-tuning and direct weight prediction baselines, yielding significant downstream performance gains. KNOW thus breaks the conventional paradigm of knowledge transfer reliant on large-scale pretraining data, establishing a new framework for post-hoc knowledge enrichment via reversible dynamics modeling.

Technology Category

• Data Mining & Knowledge Management: Knowledge Acquisition from the Web
• Machine Learning: Deep Neural Architectures and Foundation Models
• Knowledge Representation and Reasoning: Knowledge Acquisition

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Pre-trained weights have become a cornerstone of modern deep learning, enabling efficient knowledge transfer and improving downstream task performance, especially in data-scarce scenarios. However, a fundamental question remains: how can we obtain better pre-trained weights that encapsulate more knowledge beyond the given dataset? In this work, we introduce extbf{KNowledge Overflowed Weights (KNOW)} prediction, a novel strategy that leverages structured forgetting and its inversion to synthesize knowledge-enriched weights. Our key insight is that sequential fine-tuning on progressively downsized datasets induces a structured forgetting process, which can be modeled and reversed to recover knowledge as if trained on a larger dataset. We construct a dataset of weight transitions governed by this controlled forgetting and employ meta-learning to model weight prediction effectively. Specifically, our extbf{KNowledge Overflowed Weights Nowcaster (KNOWN)} acts as a hyper-model that learns the general evolution of weights and predicts enhanced weights with improved generalization. Extensive experiments across diverse datasets and architectures demonstrate that KNOW prediction consistently outperforms Naïve fine-tuning and simple weight prediction, leading to superior downstream performance. Our work provides a new perspective on reinterpreting forgetting dynamics to push the limits of knowledge transfer in deep learning.

Problem

• Predicting knowledge-enriched pre-trained weights via structured forgetting inversion
• Modeling weight transitions to recover knowledge from downsized datasets
• Enhancing generalization in deep learning through meta-learned weight prediction

Innovation

• Predicting knowledge-enriched weights via structured forgetting inversion
• Meta-learning weight transitions from controlled forgetting datasets
• Hyper-model learns weight evolution for enhanced generalization