Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

2024-12-16

AI Summary

This paper addresses the lack of a unified modeling paradigm in multimodal intelligence by proposing a next-generation multimodal learning framework centered on the unified objective of Next-Token Prediction (NTP). Methodologically, it introduces the first five-dimensional NTP taxonomy—encompassing multimodal tokenization, model architecture, task representation, datasets, and evaluation benchmarks—and integrates cross-modal tokenization, sequence-based modeling, unified prompt engineering, and multimodal benchmark construction. Key contributions include: (1) shifting multimodal learning from task-specific paradigms toward a unified objective-driven framework; (2) releasing the first open-source repository (on GitHub) dedicated to multimodal NTP, including curated literature and reproducible code; and (3) providing a systematic theoretical foundation and practical, reproducible guidelines to advance research on multimodal large language models.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Building on the foundations of language modeling in natural language processing, Next Token Prediction (NTP) has evolved into a versatile training objective for machine learning tasks across various modalities, achieving considerable success. As Large Language Models (LLMs) have advanced to unify understanding and generation tasks within the textual modality, recent research has shown that tasks from different modalities can also be effectively encapsulated within the NTP framework, transforming the multimodal information into tokens and predict the next one given the context. This survey introduces a comprehensive taxonomy that unifies both understanding and generation within multimodal learning through the lens of NTP. The proposed taxonomy covers five key aspects: Multimodal tokenization, MMNTP model architectures, unified task representation, datasets &evaluation, and open challenges. This new taxonomy aims to aid researchers in their exploration of multimodal intelligence. An associated GitHub repository collecting the latest papers and repos is available at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction

Problem

• Next Word Prediction
• Multimodal Information
• Artificial Intelligence

Innovation

• Next Token Prediction
• Large Language Models
• Multi-modal Intelligence