DSEffi-Bench: Demystifying Large Language Models' Capability in Efficient Data Science Code Generation

2026-08-31

AI Summary

该研究通过DSEffi-Bench评估大型语言模型生成数据科学代码的执行效率，揭示了仅凭正确性无法全面衡量效率，并提出了一种分类方法以指导优化。

Abstract

Current data science (DS) code generation benchmarks equate correctness with quality, overlooking execution time differences that span orders of magnitude between correct solutions. We introduce DSEffi-Bench, the first benchmark specifically targeting execution efficiency in LLM-generated DS code, comprising 1,000 instances across 10+ DS libraries with stress-testing harnesses and human-validated references. Evaluating 16 models across 3 tiers, we find that correctness alone fails to characterize efficiency: GPT-5.4 leads in correctness (Pass, 66.9\%) but its efficiency score (B$|$P, 71.7\%) nearly matches GPT-5.4-mini (71.6\%), which solves 47 fewer tasks; Kimi-K2.5 ranks lowest in correctness among frontier models (40.2\%) yet achieves the highest efficiency score (73.6\%) across all 16 models. A human-annotated five-category taxonomy reveals that 79.1\% of efficiency deficits extend beyond algorithmic complexity to domain-specific root causes, with distinct failure profiles across model tiers and libraries. Two exploratory experiments provide initial evidence that these diagnostics can guide improvement, yielding up to +14.7\% efficiency gains via taxonomy-guided optimization and approaching Claude-Opus-4.6 Best@3 in efficiency at 13.0$\times$ lower cost via library-conditioned routing.

Problem

• data science code generation
• execution efficiency
• large language models
• benchmark

Innovation

• DSEffi-Bench
• Execution Efficiency
• Data Science Code Generation
• Benchmark
• Large Language Models