Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs

2026-07-05

AI Summary

This work addresses the low inference efficiency and limited throughput of large-scale Mixture-of-Experts (MoE) language models under high-concurrency interactive deployment scenarios by proposing a multi-stage compression framework. The approach jointly optimizes heterogeneous MoE pruning, active parameter budgeting, and Mamba module pruning, integrating knowledge distillation, reinforcement learning, quantization, and multi-token prediction heads to achieve substantial model compression while preserving strong multitask performance. Experimental results demonstrate that the proposed method achieves approximately 2× higher throughput on a single 8×B200 node and increases the capacity for concurrent 1M-token requests from 1 to 8 on a single H100 GPU, with minimal degradation in downstream task accuracy.

Abstract

We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment. We designed the model to maximize server throughput under high user throughput constraints. In interactive serving workloads on a single 8xB200 node, Puzzle-75B-A9B achieves approximately 2x higher server throughput than Nemotron-3-Super at matched user throughput constraints. In ultra-long-context deployment on a single H100 GPU, the compressed model increases 1M-token concurrency from 1 request to 8 requests. Puzzle-75B-A9B is constructed using a multi-stage pipeline that combines the Iterative Puzzle compression framework with knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head. The compression process jointly optimizes heterogeneous MoE pruning, active parameter budget, and Mamba pruning to improve inference efficiency while preserving model quality. We evaluate Puzzle-75B-A9B on a broad suite of reasoning, coding, multilingual, long-context, and agentic benchmarks. Despite substantial compression, the model retains strong downstream accuracy relative to the parent model across a wide range of tasks. These results demonstrate that large hybrid MoE models can be substantially optimized for deployment efficiency while maintaining strong downstream capability.

Problem

• Hybrid MoE LLMs
• model compression
• deployment efficiency
• server throughput
• inference efficiency

Innovation

• Hybrid MoE Compression
• Iterative Puzzle Framework
• Multi-Token Prediction
• Mamba Pruning
• Deployment-Efficient LLM