EXAONE Deep: Reasoning Enhanced Language Models

2025-03-16

AI Summary

This work addresses the challenge of enhancing mathematical reasoning and code generation capabilities in small-parameter language models. We propose Long-CoT, a novel training paradigm centered on high-order reasoning, which constructs extended chain-of-thought data and jointly optimizes model performance via supervised fine-tuning on large-scale, high-quality reasoning corpora. This approach significantly improves logical deduction capacity without inflating model size. Evaluated on benchmarks for mathematical reasoning and program synthesis, EXAONE Deep—available in 2.4B and 7.8B parameter variants—outperforms mainstream models of comparable scale; its 32B variant matches top-tier open-source large language models. Key contributions include: (1) the first reasoning-specific, long-chain-of-thought data-driven training framework, balancing efficiency and effectiveness; and (2) full open release of all model weights under a commercially permissible license, enabling broad adoption in both academic research and industrial applications.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present EXAONE Deep series, which exhibits superior capabilities in various reasoning tasks, including math and coding benchmarks. We train our models mainly on the reasoning-specialized dataset that incorporates long streams of thought processes. Evaluation results show that our smaller models, EXAONE Deep 2.4B and 7.8B, outperform other models of comparable size, while the largest model, EXAONE Deep 32B, demonstrates competitive performance against leading open-weight models. All EXAONE Deep models are openly available for research purposes and can be downloaded from https://huggingface.co/LGAI-EXAONE

Problem

• Enhance language models for superior reasoning tasks
• Outperform comparable models in math and coding benchmarks
• Provide openly available models for research purposes

Innovation

• Specialized dataset for reasoning tasks
• Smaller models outperform comparable sizes
• Openly available for research purposes