Black-Box Continual Learning for Vision-Language Models

2026-06-22

AI Summary

This work addresses the challenges of continual learning with black-box vision-language models, where model weights and architecture are inaccessible, computational resources are limited, and task-agnostic inference is required. To tackle these issues, the authors propose BETA, a method that achieves efficient continual learning solely through optimization of textual prototypes. BETA comprises three key components: Semantic Projection Accumulation (SPA), Latent Distribution Replay (LDR), and Test-Time Prototype Adaptation (TTPA). The study also introduces Black-CL, the first continual learning benchmark tailored for black-box settings. Experimental results demonstrate that BETA, with only 0.05 million trainable parameters, significantly outperforms existing black-box approaches across ten datasets and achieves performance on par with or even surpassing that of white-box continual learning methods.

Abstract

The rapid deployment of Vision-Language Models (VLMs) in dynamic environments necessitates the ability to learn continuously without forgetting. However, traditional continual learning (CL) settings often rely on white-box paradigms, which is increasingly invalidated by the shift toward cloud-hosted models. In this paper, we introduce Black-CL, a more realistic benchmark for VLMs that enforces three primary real-world challenges: weight and architecture inaccessibility, constrained computation, and task-agnostic inference. The learner can query only output embeddings or logits, with no gradient flow through or structural modification of the backbone. Current CL methodologies, which rely on backbone backpropagation or complex parameter expansion, are fundamentally incompatible with these constraints. Under this setting, we propose BETA, a simple yet effective baseline built on the key insight that solely optimizing textual prototypes can navigate the complexities of CL. BETA integrates three core components: Semantic Projection Accumulation (SPA) for incremental knowledge acquisition, Latent Distribution Replay (LDR) for anchoring the embedding space against catastrophic forgetting, and Test-Time Prototype Adaptation (TTPA) for dynamic, instance-aware boundary refinement. Extensive experiments across ten diverse datasets and various backbones demonstrate that BETA significantly outperforms existing black-box tuners. Remarkably, with only 0.05 M trainable parameters, a 180--3000$\times$ reduction compared to competitive methods, BETA achieves performance on par with or even exceeding white-box CL methods. We believe Black-CL and BETA provide a foundational framework for future advancements in continual learning and accelerates the transition of continual learning from academia to real-world systems.

Problem

• Black-Box Continual Learning
• Vision-Language Models
• Catastrophic Forgetting
• Model Inaccessibility
• Task-Agnostic Inference

Innovation

• Black-box Continual Learning
• Vision-Language Models
• Textual Prototypes
• Catastrophic Forgetting
• Parameter-Efficient Learning