AlgoBench: Benchmarking Algorithmic Adaptation in Code Generation

2026-06-30

AI Summary

This work addresses the limitation of existing programming evaluation benchmarks in distinguishing whether large language models rely on memorization or possess genuine algorithmic reasoning capabilities. To this end, the authors propose AlgoBench, a framework that systematically transforms classic competitive programming problems through structured constraint migration, automatically generating new problems whose original algorithmic logic is invalidated yet remain semantically traceable, thereby compelling models to move beyond reference solutions. AlgoBench introduces novel complexity-aware evaluation metrics—such as OPTT and TRAPRATE—that prioritize algorithmic correctness over mere functional implementation. Experimental results demonstrate a significant performance drop among mainstream large language models on AlgoBench, with most ostensibly “correct” solutions failing to meet required time or space complexity constraints, thereby exposing their limited capacity for adaptive algorithmic reasoning.

Abstract

High pass rates on established programming benchmarks such as HumanEval and LiveCodeBench do not always show whether a model can reason about algorithms. Many fixed benchmarks eventually become part of the public training ecosystem through released problem statements, editorials, and generated solutions, allowing later models to improve partly by exposure rather than by stronger algorithmic ability. We introduce ALGOBENCH, a framework that automatically builds novel algorithmic problems from known competitive-programming problems through structured constraint-shifting transformations. Each accepted ALGOBENCH variant is traceable to a source problem, but must make the original reference algorithm fail. Beyond pass@$k$, we introduce complexity-aware metrics -- including OPTT, OPTS, TRAPRATE, GAPT, and CONSENS -- to test whether a solution is not only functionally correct but also asymptotically suitable for the generated problem. Experiments across multiple LLMs and prompting strategies show that performance drops sharply on ALGOBENCH variants, retrieval can increase reuse of the old algorithm, and many correct-looking solutions fail to meet the required complexity. Error analysis shows that failures are mainly algorithmic rather than implementation-level, suggesting that ALGOBENCH evaluates adaptation beyond functional correctness.

Problem

• algorithmic reasoning
• code generation
• benchmarking
• complexity analysis
• model evaluation

Innovation

• algorithmic adaptation
• code generation benchmark
• complexity-aware evaluation
• constraint-shifting transformation
• LLM reasoning