DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models

2025-07-14

AI Summary

This work addresses the core challenges of high computational cost and closed ecosystems hindering large language model (LLM) development. Methodologically, it introduces a novel LLM paradigm characterized by low cost, high performance, and open-source accessibility, achieved through an integrated approach combining multi-head latent attention (MLA), mixture-of-experts (MoE), multi-token prediction (MTP), and group relative policy optimization (GRPO), augmented by system-level engineering optimizations for end-to-end improvements in training acceleration, inference efficiency, and scalable model design. Key contributions include the open release of the DeepSeek-V3 and R1 model families, which match state-of-the-art proprietary models across multiple benchmarks while substantially reducing training and inference costs. The study systematically analyzes architectural distinctions from mainstream designs, fostering algorithm–architecture–systems co-innovation, accelerating the open LLM ecosystem, and reshaping the global AI competitive landscape.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

DeepSeek, a Chinese Artificial Intelligence (AI) startup, has released their V3 and R1 series models, which attracted global attention due to their low cost, high performance, and open-source advantages. This paper begins by reviewing the evolution of large AI models focusing on paradigm shifts, the mainstream Large Language Model (LLM) paradigm, and the DeepSeek paradigm. Subsequently, the paper highlights novel algorithms introduced by DeepSeek, including Multi-head Latent Attention (MLA), Mixture-of-Experts (MoE), Multi-Token Prediction (MTP), and Group Relative Policy Optimization (GRPO). The paper then explores DeepSeek engineering breakthroughs in LLM scaling, training, inference, and system-level optimization architecture. Moreover, the impact of DeepSeek models on the competitive AI landscape is analyzed, comparing them to mainstream LLMs across various fields. Finally, the paper reflects on the insights gained from DeepSeek innovations and discusses future trends in the technical and engineering development of large AI models, particularly in data, training, and reasoning.

Problem

• Analyzing DeepSeek's novel AI algorithms and paradigm shifts
• Exploring engineering breakthroughs in LLM scaling and optimization
• Assessing DeepSeek's impact on competitive AI landscape

Innovation

• Novel MLA and MoE algorithms enhance performance
• MTP and GRPO optimize training efficiency
• System-level architecture boosts scaling and inference