SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation

2025-05-30

AI Summary

Existing LLM code-generation benchmarks are heavily skewed toward Python, with scarce high-quality evaluation suites for Swift; HumanEval-XL and MultiPL-E suffer from translation inaccuracies and inadequate coverage of Swift-specific features (e.g., Optionals, Protocols), limiting their ability to assess models’ mastery of Swift’s core semantics. Method: We introduce SwiftEval—the first high-fidelity, human-authored benchmark tailored for Swift—comprising 28 original programming tasks, each accompanied by manually crafted, functionally rigorous test cases emphasizing language-specific challenges. Contribution/Results: Evaluating 44 state-of-the-art code LLMs reveals pronounced performance degradation on Swift-specific tasks—especially among smaller models—validating SwiftEval’s diagnostic sensitivity and benchmark rigor. SwiftEval fills a critical gap in Swift code competency assessment and establishes a new paradigm for multi-language LLM evaluation grounded in deep language semantics.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In recent years, large language models (LLMs) have showcased significant advancements in code generation. However, most evaluation benchmarks are primarily oriented towards Python, making it difficult to evaluate other programming languages, such as Swift, with high quality. By examining widely established multilingual benchmarks like HumanEval-XL and MultiPL-E, we identified critical issues specific to their Swift components, making them insufficient or even irrelevant for assessing LLM coding capabilities on Swift. Unlike these existing approaches, which prioritize rapid scaling and generalization by automatically translating Python-centric benchmarks with LLMs, we adopt a quality-over-quantity methodology. We present SwiftEval, the first Swift-oriented benchmark consisting of 28 carefully hand-crafted problems, and evaluate 44 popular Code LLMs on it. Our results show significant LLM scores drop for problems requiring language-specific features, most noticeable in the models of smaller sizes.

Problem

• Lack of high-quality benchmarks for evaluating LLM-generated Swift code
• Existing benchmarks are insufficient for assessing Swift-specific coding capabilities
• Need for a language-specific benchmark to accurately measure LLM performance in Swift

Innovation

• Hand-crafted Swift-specific benchmark problems
• Quality-over-quantity evaluation methodology
• Assessing language-specific LLM coding capabilities