Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwards

2024-05-22

AI Summary

This work addresses concept drift in multimodal large language models (MLLMs) during vision-language joint pretraining on real-world streaming data—specifically, progressive drift induced by long-tailed distributions and abrupt drift triggered by out-of-distribution samples. It is the first to systematically extend concept drift theory to cross-modal pretraining. We propose a unified multimodal concept drift modeling framework and introduce a novel T-distribution-based drift adapter that simultaneously mitigates progressive bias and explicitly models distributional shifts for robust abrupt drift detection. Coupled with dynamic image-text alignment optimization and the newly introduced long-tailed open-world multimodal dataset OpenMMlo, our approach significantly improves pretraining alignment fidelity and enhances downstream generalization in open-set, long-tailed scenarios. The code and OpenMMlo dataset are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Multi-modal Large Language Models (MLLMs) frequently face challenges from concept drift when dealing with real-world streaming data, wherein distributions change unpredictably. This mainly includes gradual drift due to long-tailed data and sudden drift from Out-Of-Distribution (OOD) data, both of which have increasingly drawn the attention of the research community. While these issues have been extensively studied in the individual domain of vision or language, their impacts on MLLMs in concept drift settings remain largely underexplored. In this paper, we reveal the susceptibility and vulnerability of Vision-Language (VL) models to significant biases arising from gradual drift and sudden drift, particularly in the pre-training. To effectively address these challenges, we propose a unified framework that extends concept drift theory to the multi-modal domain, enhancing the adaptability of the VL model to unpredictable distribution changes. Additionally, a T-distribution based drift adapter is proposed to effectively mitigate the bias induced by the gradual drift, which also facilitates the model in distinguishing sudden distribution changes through explicit distribution modeling. Extensive experiments demonstrate our method enhances the efficiency and accuracy of image-text alignment in the pre-training of VL models, particularly in the concept drift scenario. Moreover, various downstream tasks exhibit significant improvements in our model's ability to adapt to the long-tailed open world. Furthermore, we create a set of multi-modal datasets called OpenMMlo, specifically tailored for the long-tailed open-world setting, to validate our findings. To foster the development of the multi-modal community, we have made both OpenMMlo datasets and our code publicly available at: https://github.com/XiaoyuYoung/ConceptDriftMLLMs.

Problem

• Adapting MLLMs to concept drift
• Mitigating biases from gradual and sudden drift
• Enhancing VL models' adaptability to distribution changes

Innovation

• Unified multi-modal concept drift framework
• T-distribution based drift adapter
• OpenMMlo datasets for validation