EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization

2024-05-24

Neural Information Processing Systems

AI Summary

To address the high latency and memory overhead of code generated by large language models (LLMs), this paper proposes the first execution-cost-driven closed-loop self-optimization framework: generate → local execution profiling (time/memory) → structured feedback → LLM-based iterative rewriting. It requires no human annotation or model fine-tuning, enabling general-purpose code efficiency improvement. We introduce EffiBench, a novel efficiency evaluation benchmark, and design execution-bottleneck-aware prompt engineering alongside a multi-model collaborative optimization mechanism. Evaluated on StarCoder2-15B, our approach reduces average task execution time by 87.1% and total memory consumption by 90.8%. Moreover, it significantly improves runtime efficiency across 16 open-source and 6 closed-source LLMs, demonstrating broad applicability and effectiveness in optimizing generated code for real-world resource constraints.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models (LLMs) have shown remarkable progress in code generation, but their generated code often suffers from inefficiency, resulting in longer execution times and higher memory consumption. To address this issue, we propose extbf{EffiLearner}, a self-optimization framework that utilizes execution overhead profiles to improve the efficiency of LLM-generated code. EffiLearner first generates code using an LLM, then executes it locally to capture execution time and memory usage profiles. These profiles are fed back to the LLM, which then revises the code to reduce overhead. To evaluate the effectiveness of EffiLearner, we conduct extensive experiments on the EffiBench, HumanEval, and MBPP with 16 open-source and 6 closed-source models. Our evaluation results demonstrate that through iterative self-optimization, EffiLearner significantly enhances the efficiency of LLM-generated code. For example, the execution time (ET) of StarCoder2-15B for the EffiBench decreases from 0.93 (s) to 0.12 (s) which reduces 87.1% the execution time requirement compared with the initial code. The total memory usage (TMU) of StarCoder2-15B also decreases from 22.02 (Mb*s) to 2.03 (Mb*s), which decreases 90.8% of total memory consumption during the execution process. The source code of EffiLearner was released in https://github.com/huangd1999/EffiLearner

Problem

• Improving efficiency of LLM-generated code
• Reducing execution time and memory usage
• Self-optimizing code via execution profiles

Innovation

• Self-optimization framework for code efficiency
• Uses execution profiles to refine LLM-generated code
• Iterative feedback reduces runtime and memory usage