Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning

2025-10-19

AI Summary

Pretrained models suffer from inefficient similarity matching in continual representation learning due to feature multicollinearity, while state-of-the-art methods incur high computational overhead, hindering low-latency deployment. To address this, we propose Fly-CL—the first continual learning framework inspired by the Drosophila olfactory circuit. Fly-CL achieves feature decorrelation under a near-frozen backbone via sparse projection, random mapping, and a biologically interpretable dimensional expansion mechanism. We theoretically prove that it asymptotically eliminates multicollinearity with extremely low time complexity. Fly-CL is architecture- and dataset-agnostic, significantly reducing training latency while matching or exceeding SOTA performance across diverse benchmarks. The implementation is publicly available.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search
• Computer Vision: Representation Learning for Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Using a nearly-frozen pretrained model, the continual representation learning paradigm reframes parameter updates as a similarity-matching problem to mitigate catastrophic forgetting. However, directly leveraging pretrained features for downstream tasks often suffers from multicollinearity in the similarity-matching stage, and more advanced methods can be computationally prohibitive for real-time, low-latency applications. Inspired by the fly olfactory circuit, we propose Fly-CL, a bio-inspired framework compatible with a wide range of pretrained backbones. Fly-CL substantially reduces training time while achieving performance comparable to or exceeding that of current state-of-the-art methods. We theoretically show how Fly-CL progressively resolves multicollinearity, enabling more effective similarity matching with low time complexity. Extensive simulation experiments across diverse network architectures and data regimes validate Fly-CL's effectiveness in addressing this challenge through a biologically inspired design. Code is available at https://github.com/gfyddha/Fly-CL.

Problem

• Resolving multicollinearity in similarity matching for continual learning
• Reducing computational complexity for real-time representation learning
• Mitigating catastrophic forgetting in pretrained model-based continual learning

Innovation

• Fly-CL uses bio-inspired design for efficient decorrelation
• It progressively resolves multicollinearity in similarity matching
• Framework reduces training time while maintaining high performance