RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks

2024-11-25

arXiv.org

AI Summary

This paper addresses incremental learning under resource-constrained settings, proposing an efficient, low-parameter method for adapting pretrained models to novel classes. The approach comprises three key contributions: (1) a Neural Mimicry mechanism that reconstructs pretrained weights with high fidelity; (2) a soft parameter-sharing framework based on weight decomposition, enabling layer-wise reparameterization using fewer than 50 task-specific parameters per layer; and (3) a frozen-backbone architecture augmented with learnable scaling coefficients, eliminating the need for fine-tuning or retraining the base model. The method is model-agnostic—compatible with arbitrary architectures and scales—and integrates seamlessly with existing techniques. Evaluated on six benchmark datasets, it outperforms state-of-the-art methods by 3% in accuracy while reducing parameter count by several orders of magnitude compared to LoRA. This substantial parameter efficiency significantly enhances feasibility for deployment on edge and mobile devices.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Incremental learning aims to adapt to new sets of categories over time with minimal computational overhead. Prior work often addresses this task by training efficient task-specific adaptors that modify frozen layer weights or features to capture relevant information without affecting predictions on previously learned categories. While these adaptors are generally more efficient than finetuning the entire network, they still require tens to hundreds of thousands of task-specific trainable parameters even for relatively small networks, making it challenging to operate on resource-constrained environments with high communication costs like edge devices or mobile phones. Thus, we propose Reparameterized, Compact weight Adaptation for Sequential Tasks (RECAST), a novel method that dramatically reduces task-specific trainable parameters to fewer than 50 - several orders of magnitude less than competing methods like LoRA. RECAST accomplishes this efficiency by learning to decompose layer weights into a soft parameter-sharing framework consisting of shared weight templates and very few module-specific scaling factors or coefficients. This soft parameter-sharing framework allows for effective task-wise reparameterization by tuning only these coefficients while keeping templates frozen.A key innovation of RECAST is the novel weight reconstruction pipeline called Neural Mimicry, which eliminates the need for pretraining from scratch. This allows for high-fidelity emulation of existing pretrained weights within our framework and provides quick adaptability to any model scale and architecture. Extensive experiments across six datasets demonstrate RECAST outperforms the state-of-the-art by up to 3% across various scales, architectures, and parameter spaces Moreover, we show that RECAST's architecture-agnostic nature allows for seamless integration with existing methods, further boosting performance.

Problem

• Reduces task-specific trainable parameters for incremental learning.
• Enables efficient adaptation on resource-constrained devices like edge devices.
• Introduces Neural Mimicry for high-fidelity emulation of pretrained weights.

Innovation

• Reduces task-specific parameters significantly
• Uses shared weight templates and coefficients
• Introduces Neural Mimicry for weight reconstruction