Apriel-Nemotron-15B-Thinker

2025-08-13

AI Summary

To address the prohibitively high memory and computational overhead of large language models (LLMs) in enterprise deployment, this paper proposes an efficient inference model construction paradigm. Our method comprises a four-stage training pipeline: base model expansion, continued pretraining, supervised fine-tuning, and GRPO-based reinforcement learning—yielding a 15B-parameter model that matches or exceeds the performance of 32B baselines. Experiments demonstrate a 50% reduction in memory footprint and significantly improved inference latency, while maintaining or surpassing state-of-the-art models—including o1-mini and QWQ-32B—on diverse benchmarks spanning code generation and mathematical reasoning. This work empirically validates the feasibility of “small-parameter, high-efficiency” LLM architectures for enterprise applications, providing a reproducible technical pathway and empirical evidence for lightweight LLM deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

While large language models (LLMs) have achieved remarkable reasoning capabilities across domains like code, math and other enterprise tasks, their significant memory and computational costs often preclude their use in practical enterprise settings. To this end, we introduce Apriel-Nemotron-15B-Thinker, a 15-billion parameter model in the ServiceNow Apriel SLM series that achieves performance against medium sized state-of-the-art models such as o1-mini, QWQ32B, and EXAONE-Deep-32B while maintaining only half the memory footprint of those alternatives. Apriel-Nemotron-15B-Thinker model is trained in a four stage training pipeline including 1) Base Model upscaling, 2) Continual Pre-training 3) Supervised Fine-tuning (SFT) and 4) Reinforcement Learning using GRPO. Comprehensive evaluations across a diverse suite of benchmarks consistently demonstrate that our Apriel-Nemotron-15B-Thinker model matches or exceeds the performance of its 32-billion parameter counterparts, despite being less than half their size.

Problem

• Reducing memory and computational costs of large language models
• Matching performance of larger models with smaller size
• Optimizing model training pipeline for enterprise efficiency

Innovation

• 15B parameter model with half memory footprint
• Four-stage training pipeline for efficiency
• Matches performance of larger 32B models