Magistral

2025-06-12

AI Summary

This work investigates end-to-end training of large language models (LLMs) under a pure reinforcement learning (RL) paradigm, aiming to preserve and enhance multimodal understanding, instruction following, and function-calling capabilities—using text-only data. To this end, we design a scalable, fully self-contained RL training pipeline built entirely on custom infrastructure, without reliance on external RL trajectories or knowledge distillation data. We introduce the first text-driven RL training method enabling fine-grained, language-guided control over reasoning steps. Leveraging Mistral Medium 3 as the base model, we devise cold-start dataset construction and policy optimization algorithms. We release Magistral Medium—a pure-RL inference model—and open-source Magistral Small—including the cold-start dataset—under the Apache 2.0 license. Experiments demonstrate substantial gains over supervised fine-tuning baselines on complex reasoning benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We introduce Magistral, Mistral's first reasoning model and our own scalable reinforcement learning (RL) pipeline. Instead of relying on existing implementations and RL traces distilled from prior models, we follow a ground up approach, relying solely on our own models and infrastructure. Notably, we demonstrate a stack that enabled us to explore the limits of pure RL training of LLMs, present a simple method to force the reasoning language of the model, and show that RL on text data alone maintains most of the initial checkpoint's capabilities. We find that RL on text maintains or improves multimodal understanding, instruction following and function calling. We present Magistral Medium, trained for reasoning on top of Mistral Medium 3 with RL alone, and we open-source Magistral Small (Apache 2.0) which further includes cold-start data from Magistral Medium.

Problem

• Develop scalable RL pipeline for LLM training
• Explore pure RL training limits for LLMs
• Maintain capabilities with RL on text data

Innovation

• Scalable pure RL training pipeline
• Method to force reasoning language
• RL maintains initial model capabilities