Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

2025-03-03

AI Summary

To address capability bottlenecks of small-scale multimodal language models in mathematical reasoning, code generation, multilingual support, and long-context modeling, this work introduces Phi-4-Mini (3.8B) and its unified multimodal extension, Phi-4-Multimodal. Methodologically, we propose: (1) a novel modality-specific LoRA router enabling interference-free joint inference over text, vision, and speech; (2) a 200K-token vocabulary integrated with grouped-query attention (GQA) to enhance multilingual coverage and long-sequence efficiency; and (3) a training pipeline combining high-quality synthetic data distillation, MoE-style LoRA adaptation, and multi-stage instruction tuning. Experiments demonstrate that Phi-4-Mini surpasses same-parameter open-source models on mathematical and coding benchmarks—matching the performance of 7B-class models. Phi-4-Multimodal achieves state-of-the-art cross-modal reasoning and outperforms larger models on multimodal benchmarks including OpenASR, establishing new efficiency–capability trade-offs for compact multimodal architectures.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language model trained on high-quality web and synthetic data, significantly outperforming recent open-source models of similar size and matching the performance of models twice its size on math and coding tasks requiring complex reasoning. This achievement is driven by a carefully curated synthetic data recipe emphasizing high-quality math and coding datasets. Compared to its predecessor, Phi-3.5-Mini, Phi-4-Mini features an expanded vocabulary size of 200K tokens to better support multilingual applications, as well as group query attention for more efficient long-sequence generation. Phi-4-Multimodal is a multimodal model that integrates text, vision, and speech/audio input modalities into a single model. Its novel modality extension approach leverages LoRA adapters and modality-specific routers to allow multiple inference modes combining various modalities without interference. For example, it now ranks first in the OpenASR leaderboard to date, although the LoRA component of the speech/audio modality has just 460 million parameters. Phi-4-Multimodal supports scenarios involving (vision + language), (vision + speech), and (speech/audio) inputs, outperforming larger vision-language and speech-language models on a wide range of tasks. Additionally, we experiment to further train Phi-4-Mini to enhance its reasoning capabilities. Despite its compact 3.8-billion-parameter size, this experimental version achieves reasoning performance on par with or surpassing significantly larger models, including DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B.

Problem

• Develop compact, high-performance multimodal language models.
• Enhance reasoning and multilingual support in small-scale models.
• Integrate multiple input modalities efficiently without performance loss.

Innovation

• Uses Mixture-of-LoRAs for multimodal integration
• Expands vocabulary to 200K tokens for multilingual support
• Enhances reasoning with curated synthetic data