Mashup Learning: Faster Finetuning by Remixing Past Checkpoints

2026-03-10

AI Summary

This work addresses the inefficiency and resource waste caused by the proliferation of isolated, domain-specific fine-tuned model checkpoints, which hinder rapid adaptation to new tasks. It proposes a novel and systematic transfer learning paradigm that treats historical fine-tuning checkpoints as reusable assets. The approach first evaluates the relevance of existing checkpoints to the target task, selects the most pertinent ones, and fuses their parameters to construct an improved initialization for subsequent standard fine-tuning. Extensive experiments across eight benchmarks and four backbone architectures demonstrate that this method consistently enhances model accuracy by 0.5–5.0 percentage points, reduces the required training steps by 41–46%, and lowers total training time by up to 37%.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Finetuning on domain-specific data is a well-established method for enhancing LLM performance on downstream tasks. Training on each dataset produces a new set of model weights, resulting in a multitude of checkpoints saved in-house or on open-source platforms. However, these training artifacts are rarely reused for subsequent experiments despite containing improved model abilities for potentially similar tasks. In this paper, we propose Mashup Learning, a simple method to leverage the outputs of prior training runs to enhance model adaptation to new tasks. Our procedure identifies the most relevant historical checkpoints for a target dataset, aggregates them with model merging, and uses the result as an improved initialization for training. Across 8 standard LLM benchmarks, four models, and two collections of source checkpoints, Mashup Learning consistently improves average downstream accuracy by 0.5-5 percentage points over training from scratch. It also accelerates convergence, requiring 41-46% fewer training steps and up to 37% less total wall-clock time to match from-scratch accuracy, including all selection and merging overhead.

Problem

• finetuning
• model checkpoints
• LLM adaptation
• training efficiency
• knowledge reuse

Innovation

• Mashup Learning
• model merging
• checkpoint reuse
• faster finetuning
• LLM adaptation