MODL: Multilearner Online Deep Learning

2024-05-28

arXiv.org

AI Summary

Online deep learning faces the fundamental challenge of balancing “speed” (real-time model updates) and “depth” (strong representation capacity). To address this, we propose a hybrid multi-learner architecture: a closed-form recursive logistic regression learner—free of backpropagation—enables millisecond-level incremental updates; representation depth is enhanced via cascaded shallow and deep learners coupled with a collaborative training mechanism. Our key innovation is the first gradient-free closed-form recursive parameter update method, alongside a unified optimization framework for jointly optimizing speed and depth in online multi-learner systems. Extensive evaluation on standard data stream benchmarks demonstrates significant improvements over state-of-the-art methods, achieving new SOTA performance. We release a fully open-sourced implementation to foster reproducibility and further research.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Online deep learning tackles the challenge of learning from data streams by balancing two competing goals: fast learning and deep learning. However, existing research primarily emphasizes deep learning solutions, which are more adept at handling the ``deep'' aspect than the ``fast'' aspect of online learning. In this work, we introduce an alternative paradigm through a hybrid multilearner approach. We begin by developing a fast online logistic regression learner, which operates without relying on backpropagation. It leverages closed-form recursive updates of model parameters, efficiently addressing the fast learning component of the online learning challenge. This approach is further integrated with a cascaded multilearner design, where shallow and deep learners are co-trained in a cooperative, synergistic manner to solve the online learning problem. We demonstrate that this approach achieves state-of-the-art performance on standard online learning datasets. We make our code available: https://github.com/AntonValk/MODL

Problem

• Balancing fast and deep learning in data streams
• Integrating shallow and deep learners synergistically
• Achieving state-of-the-art online learning performance

Innovation

• Hybrid multilearner approach for online learning
• Fast logistic regression without backpropagation
• Cascaded co-training of shallow and deep learners