Every Sample Matters: Leveraging Mixture-of-Experts and High-Quality Data for Efficient and Accurate Code LLM

2025-03-22

AI Summary

To address the challenge of balancing performance and inference efficiency in large language models for code, this paper introduces Ling-Coder-Lite—a lightweight, high-efficiency code LLM. Methodologically, it pioneers a synergistic optimization paradigm integrating program analysis—both static and dynamic—for high-quality data curation with a Mixture-of-Experts (MoE) sparse architecture, complemented by targeted post-training strategies. Experiments demonstrate that Ling-Coder-Lite matches the performance of Qwen2.5-Coder-7B and DeepSeek-Coder-V2-Lite across 12 mainstream programming benchmarks, while reducing inference latency by 42%, increasing throughput by 1.8×, and cutting deployment resource requirements by 50% at equivalent performance. Both the full model weights and the curated high-quality code dataset are publicly released. The core contribution is a program-semantics-aware joint optimization framework unifying data selection and architectural design, establishing a new paradigm for high-performance code intelligence under resource constraints.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent advancements in code large language models (LLMs) have demonstrated remarkable capabilities in code generation and understanding. It is still challenging to build a code LLM with comprehensive performance yet ultimate efficiency. Many attempts have been released in the open source community to break the trade-off between performance and efficiency, such as the Qwen Coder series and the DeepSeek Coder series. This paper introduces yet another attempt in this area, namely Ling-Coder-Lite. We leverage the efficient Mixture-of-Experts (MoE) architecture along with a set of high-quality data curation methods (especially those based on program analytics) to build an efficient yet powerful code LLM. Ling-Coder-Lite exhibits on-par performance on 12 representative coding benchmarks compared to state-of-the-art models of similar size, such as Qwen2.5-Coder-7B and DeepSeek-Coder-V2-Lite, while offering competitive latency and throughput. In practice, we achieve a 50% reduction in deployment resources compared to the similar-sized dense model without performance loss. To facilitate further research and development in this area, we open-source our models as well as a substantial portion of high-quality data for the annealing and post-training stages. The models and data can be accessed at~url{https://huggingface.co/inclusionAI/Ling-Coder-lite}.

Problem

• Balancing performance and efficiency in code LLMs
• Leveraging MoE and high-quality data for better models
• Reducing deployment resources without performance loss

Innovation

• Utilizes Mixture-of-Experts (MoE) architecture
• Employs high-quality data curation methods
• Reduces deployment resources by 50%