Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

2026-06-12

AI Summary

This work addresses the limitations of large language models in efficiency and capability when handling long-context, high-throughput, and complex reasoning tasks. The authors propose a 55B-total-parameter (5.5B active-parameter) Mixture-of-Experts architecture that uniquely integrates LatentMoE, a hybrid Mamba-Attention mechanism, multi-token prediction, NVFP4 quantization-aware pretraining, multi-environment Reinforcement Learning with Value Rescaling (RLVR), and online multi-teacher distillation. Pretrained on 20 trillion tokens and scaled to a 1M-token context length, the model achieves approximately six times the inference throughput of current state-of-the-art open-source models while maintaining comparable accuracy, substantially enhancing performance on long-horizon autonomous agent tasks. The code, data, and training protocols have been publicly released.

Abstract

We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation (MOPD). Nemotron 3 Ultra is our most capable model yet, employing multiple key technologies - LatentMoE, Multi Token Prediction (MTP), NVFP4 pre-training, multi-environment RLVR, MOPD, and reasoning budget control. Nemotron 3 Ultra achieves up to ~6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy. The state-of-the-art accuracy, high inference throughput, and 1M token context length make Nemotron 3 Ultra ideal for long-running autonomous agentic tasks. We open-source the base, post-trained, and quantized checkpoints, along with the training data and recipe on HuggingFace.

Problem

• agentic reasoning
• Mixture-of-Experts
• long context
• inference throughput
• large language model

Innovation

• Mixture-of-Experts
• Hybrid Mamba-Transformer
• Multi Token Prediction
• Reinforcement Learning with Value Ranking
• On-Policy Distillation