CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis

2025-03-29

AI Summary

Existing LLM-based agents for inductive program synthesis suffer from static, feedback-free, and context-agnostic evaluation protocols—particularly misaligned with real-world scenarios like reverse engineering. Method: We propose CodeARC, the first large-scale interactive benchmark for program synthesis, comprising 1,114 generalizable functions and introducing a closed-loop paradigm: “query–synthesis–differential-testing feedback–iterative refinement,” supporting function invocation and self-correction. Crucially, it establishes the first interactive evaluation framework grounded in differential-testing oracles, moving beyond static I/O examples and held-out test sets. Contribution/Results: Experiments show o3-mini achieves the highest success rate (52.7%) on CodeARC; LLaMA-3.1-8B-Instruct attains a 31% relative improvement after fine-tuning on synthetic trajectories. These results empirically validate that interactive evaluation more effectively captures—and rigorously challenges—reasoning capabilities in program synthesis.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Inductive program synthesis, or programming by example, requires synthesizing functions from input-output examples that generalize to unseen inputs. While large language model agents have shown promise in programming tasks guided by natural language, their ability to perform inductive program synthesis is underexplored. Existing evaluation protocols rely on static sets of examples and held-out tests, offering no feedback when synthesized functions are incorrect and failing to reflect real-world scenarios such as reverse engineering. We propose CodeARC, the Code Abstraction and Reasoning Challenge, a new evaluation framework where agents interact with a hidden target function by querying it with new inputs, synthesizing candidate functions, and iteratively refining their solutions using a differential testing oracle. This interactive setting encourages agents to perform function calls and self-correction based on feedback. We construct the first large-scale benchmark for general-purpose inductive program synthesis, featuring 1114 functions. Among 18 models evaluated, o3-mini performs best with a success rate of 52.7%, highlighting the difficulty of this task. Fine-tuning LLaMA-3.1-8B-Instruct on curated synthesis traces yields up to a 31% relative performance gain. CodeARC provides a more realistic and challenging testbed for evaluating LLM-based program synthesis and inductive reasoning.

Problem

• Evaluating LLM agents' inductive program synthesis from examples
• Addressing limitations of static tests with interactive feedback
• Benchmarking 1114 functions for realistic program synthesis scenarios

Innovation

• Interactive evaluation framework with hidden functions
• Differential testing oracle for iterative refinement
• Large-scale benchmark with 1114 functions