Constraint-based Pre-training: From Structured Constraints to Scalable Model Initialization

2026-04-16

AI Summary

This work addresses the inflexibility of conventional pre-trained models, whose fixed sizes hinder adaptation to downstream tasks requiring varying model scales. The authors propose a novel pre-training paradigm based on structured constraints, introducing Kronecker factorization into pre-training for the first time. This approach decouples model weights into a scale-invariant, reusable weight template and a lightweight, data-driven weight scaler, framing variable-scale model initialization as a multi-task adaptation problem. The method supports arbitrary depths and widths in both Transformer and CNN architectures, significantly accelerating convergence and improving performance across diverse tasks—including image classification, generation, and embodied control—thereby enabling efficient and flexible model deployment.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

The pre-training and fine-tuning paradigm has become the dominant approach for model adaptation. However, conventional pre-training typically yields models at a fixed scale, whereas practical deployment often requires models of varying sizes, exposing its limitations when target model scales differ from those used during pre-training. To address this, we propose an innovative constraint-based pre-training paradigm that imposes structured constraints during pre-training to disentangle size-agnostic knowledge into reusable weight templates, while assigning size-specific adaptation to lightweight weight scalers, thereby reformulating variable-sized model initialization as a multi-task adaptation problem. Within this paradigm, we further introduce WeiT, which employs Kronecker-based constraints to regularize the pre-training process. Specifically, model parameters are represented as compositions of weight templates via concatenation and weighted aggregation, with adaptive connections governed by lightweight weight scalers whose parameters are learned from limited data. This design enables flexible and efficient construction of model weights across diverse downstream scales. Extensive experiments demonstrate the efficiency and effectiveness of WeiT, achieving state-of-the-art performance in initializing models with varying depths and widths across a broad range of perception and embodied learning tasks, including Image Classification, Image Generation, and Embodied Control. Moreover, its effectiveness generalizes to both Transformer-based and Convolution-based architectures, consistently enabling faster convergence and improved performance even under full training.

Problem

• pre-training
• model scaling
• variable-sized models
• model initialization
• transfer learning

Innovation

• constraint-based pre-training
• weight templates
• weight scalers
• Kronecker constraints
• scalable model initialization