Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models

2026-04-14

AI Summary

This work addresses the inefficiency and trade-offs between performance and computational cost when scaling traditional vision foundation models into multi-scale model families. The authors propose Chain-of-Models Pre-Training (CoM-PT), a scalable acceleration framework operating at the model-family level: only the smallest model undergoes standard pre-training, while larger models are efficiently trained via inverse knowledge transfer in both parameter and feature spaces, achieving comparable or superior performance without additional computational overhead. CoM-PT is compatible with diverse pre-training paradigms, matches or exceeds independently trained models across 45 datasets, reduces computational costs by up to 72% on CC3M, and achieves a 7.09× overall training speedup when scaling to a family of seven models—demonstrating the principle that “training more yields greater efficiency.”

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

In this paper, we present Chain-of-Models Pre-Training (CoM-PT), a novel performance-lossless training acceleration method for vision foundation models (VFMs). This approach fundamentally differs from existing acceleration methods in its core motivation: rather than optimizing each model individually, CoM-PT is designed to accelerate the training pipeline at the model family level, scaling efficiently as the model family expands. Specifically, CoM-PT establishes a pre-training sequence for the model family, arranged in ascending order of model size, called model chain. In this chain, only the smallest model undergoes standard individual pre-training, while the other models are efficiently trained through sequential inverse knowledge transfer from their smaller predecessors by jointly reusing the knowledge in the parameter space and the feature space. As a result, CoM-PT enables all models to achieve performance that is mostly superior to standard individual training while significantly reducing training cost, and this is extensively validated across 45 datasets spanning zero-shot and fine-tuning tasks. Notably, its efficient scaling property yields a remarkable phenomenon: training more models even results in higher efficiency. For instance, when pre-training on CC3M: i) given ViT-L as the largest model, progressively prepending smaller models to the model chain reduces computational complexity by up to 72%; ii) within a fixed model size range, as the VFM family scales across 3, 4, and 7 models, the acceleration ratio of CoM-PT exhibits a striking leap: from 4.13X to 5.68X and 7.09X. Since CoM-PT is naturally agnostic to specific pre-training paradigms, we open-source the code to spur further extensions in more computationally intensive scenarios, such as large language model pre-training.

Problem

• training acceleration
• vision foundation models
• pre-training
• model scaling
• computational efficiency

Innovation

• Chain-of-Models
• training acceleration
• vision foundation models
• inverse knowledge transfer
• model family scaling