Performance Evaluation of General Purpose Large Language Models for Basic Linear Algebra Subprograms Code Generation

2025-07-07

AI Summary

This work addresses the feasibility of leveraging state-of-the-art general-purpose large language models (LLMs)—specifically GPT-4.1 and o4-mini—for automated, high-performance BLAS (Levels 1–3) C code generation directly on CPU platforms. Method: Given only standard routine names or reference Fortran implementations, we employ prompt engineering to elicit LLM outputs incorporating OpenMP parallelism, SIMD vectorization, and cache-blocking optimizations, followed by rigorous functional correctness verification and performance benchmarking against BLAS specifications. Contribution/Results: The LLMs generate syntactically valid and semantically compliant code without natural-language descriptions; approximately 70% of routines pass functional tests upon first-generation. Notably, several LLM-generated implementations surpass OpenBLAS in single- and multi-threaded performance, achieving up to 2.3× speedup. This study demonstrates, for the first time, the practical viability of general-purpose LLMs in generating production-grade numerical library kernels, establishing a novel paradigm for AI-assisted development of high-performance computing infrastructure.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Generative AI technology based on Large Language Models (LLM) has been developed and applied to assist or automatically generate program codes. In this paper, we evaluate the capability of existing general LLMs for Basic Linear Algebra Subprograms (BLAS) code generation for CPUs. We use two LLMs provided by OpenAI: GPT-4.1, a Generative Pre-trained Transformer (GPT) model, and o4-mini, one of the o-series of Reasoning models. Both have been released in April 2025. For the routines from level-1 to 3 BLAS, we tried to generate (1) C code without optimization from routine name only, (2) C code with basic performance optimizations (thread parallelization, SIMD vectorization, and cache blocking) from routine name only, and (3) C code with basic performance optimizations based on Fortran reference code. As a result, we found that correct code can be generated in many cases even when only routine name are given. We also confirmed that thread parallelization with OpenMP, SIMD vectorization, and cache blocking can be implemented to some extent, and that the code is faster than the reference code.

Problem

• Evaluate LLMs for BLAS code generation on CPUs
• Assess GPT-4.1 and o4-mini for optimized C code
• Compare generated code performance with Fortran reference

Innovation

• Evaluates GPT-4.1 and o4-mini for BLAS code generation
• Generates optimized C code from routine names
• Implements OpenMP, SIMD, and cache blocking optimizations