Yi: Open Foundation Models by 01.AI

2024-03-07

arXiv.org

AI Summary

This work addresses key challenges in multimodal large language models (MLLMs): long-context understanding, vision-language alignment, and data quality control. Methodologically, it introduces the open-source Yi series foundation models (6B/34B), encompassing variants for general language modeling, dialogue, 200K-context processing, deep extension, and vision-language integration (ViT-LLM). A novel three-tier data quality framework is proposed: (1) 3.1 trillion tokens of bilingual, deduplicated pretraining data; (2) a high-fidelity, human-annotated fine-tuning set (<10K samples); and (3) lightweight continual pretraining to extend context length and enhance capabilities. Techniques include cascaded data filtering, multi-round human verification, and explicit vision-language alignment mechanisms. Experimental results demonstrate state-of-the-art performance: the base model achieves SOTA on MMLU; the dialogue variant leads in human preference win rates on AlpacaEval and Chatbot Arena; and the 200K-context variant significantly improves needle-in-a-haystack retrieval accuracy.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.

Problem

• Natural Language Processing
• Image Recognition
• Multimodal Information Processing

Innovation

• Multimodal Understanding
• Transformer Architecture
• Depth Learning